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APPARATUS AND METHOD FOR ALIGNMENT OF 
SPATIAL OR TEMPORAL NON-OVERLAPPING IMAGE SEQUENCES 

FIELD OF THE INVENTION 
. The present invention relates to methodologies and systems for combining 
and integrating visual information from various sources. 

BACKGROUND OF THE INVENTION 

The problem of image alignment or image registration has been extensively 
researched and successful methods for solving this problem have been developed. Some 
of these methods are based on matching extracted local image features. Other 
approaches are based on directly matching image intensities. However, all these 
methods share one basic assumption: that there is sufficient overlap between the images 
to allow extraction of common image properties, namely, that there is sufficient 
"similarity" between the images. The term similarity is used here in the broadest sense, 
for example: gray-level similarity, feature similarity, similarity of frequencies and 
statistical similarity, such as mutual information. Consequently, the prior art does not 
provide alignment between images where there is very little similarity between the 
images or where there is no spatial overlap between the images-. 

However, a sequence of images contains much more information than any 
individual image does. For example: a sequence of images of a scene contains temporal 
changes caused by dynamic changes in the scene, or temporal changes induced by the 
motion of the camera. This information is employed by the present invention to provide 
alignment between images of the two sequences of images even when the sequences 
have no spatial overlap or have very low similarity. 

State of the art methods for sequence-to-sequence alignment, and other 
relevant technologies, are described in: 

[1] S. Avidan and A. Shashua. Threading fundamental matrices. In European 
Conference on Computer Vision, 1998. 
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[2] P. A. Beardsley, P. H. S. Torr and A. Zisserman. 3D model acquisition from 
extended image sequences. In Proc. 4 th European Conference on Computer Vision, 
LNCS 1065, Cambridge, pages 683-695, 1996. 

[3] P. R. Burt and R. J. Kolczynski. Enhanced image capture through fusion. In 
5 International Conference on Computer Vision, 1993, 

[4] Y. Caspt and M Irani. A step towards sequence-to-sequence alignment In IEEE 
Conference on Computer Vision and Pattern Recognition, Hilton Head Island, South 
Carolina, June 2000. 

[5] D. Demirdijian, A. Zisserman and R. Horaud. Stereo autocalibration from one plane. 
10 In European Conference on Computer Vision, 2000. 

[6] Y. Dufournaud, G. * Schmid and R: Horaud. Matching images with different 
resolutions. In IEEE Conference on Computer Vision and Pattern Recognition, Hilton 
Head Island, South Carolina, June 2000. 

[7] C. E. Pearson (ed}. Handbook of applied mathematics - Second Edition. Van 
1 5 Nostrand Reinhold Company, New York, .1 983, pp 898. 

[8] F. R. Gantmakher. The theory of matrices. Chapter VIII, Chelsea Pub;, New York, 
1959. • 

[9] Gene Golub and Charles Van Loan. Matrix Computations. The Johns Hopkins 
University Press,Baltimore and London, pp. 123-127, 1989 
20 [1 0] Richard 1. Hartley. In Defence of the 8-point Algorithm. In Pattern Recognition and 
Machine Intelligence 19 (6) June pages 580-593 1997 

[11] R. Hartley and A. Zisserman.- Multiple View Geometry in Computer Vision. 
Chapter 3, Cambridge university press, Cambridge, 2000. 

[12] R. Horaud and G. Csurka. Reconstruction using motions of a stereo rig. In 
25 International Conference on Computer Vision, pages 96-103, 1998. 

[13] R. Horaud and F. Dornaika. Hand-eye calibration. International Journal of Robotics 
Research, 14(3): 195-210, June 1995. 

[14] M, Irani and P. Anandan. About direct methods. In Vision Algorithms Workshop, 
pages 267-277, Corfu, 1999. 
30 [15] M. Irani, P. Anandan and D. Weinshall. From reference frames to reference planes: 
Multi-view parallax geometry and applications. In European Conference on Computer 
Vision, Freiburg, June 1998. 
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[16] M. Irani, B. Rousso and S. Peleg. Computing occluding and transparent motions. 
International Journal of Computer Vision, 12(1):5-16, January 1994. 
[17] R. Kumar, P. Anandan and K. Hanna. Direct recovery of shape from multiple 
views: parallax based approach. In International Conference on Pattern Recognition, 
5 1994. 

[18] Harpreet Sawhney. 3D geometry from planar parallax. In IEEE Conference on 
Computer Vision and Pattern Recognition, June 1 994. 

[] 9] A. Shashua and N. Navab. Relative affine structure: Theory and application to 3D 
reconstruction from perspective views. In IEEE Conference on Computer Vision and 
10 Pattern Recognition, pages 483-489, Seattle, WA, June 1994. • 

[20] G. P. Stein. Tracking from multiple view points: Self-calibration-of space and time. . 
In DARPA IU Workshop, pages 1037-1042, 1998. 

[21] P. H. S. Torr and A. Zissennan. Feature based methods for structure and motion 
estimation. In Vision Algorithms Workshop, pages 279-329, Corfu, 1999. 
1 5 [22] R. Y. Tsai and R. K. Lenz. A new technique for full autonomous and efficient. 3D 
robotics hand/eye calibration. IEEE Journal of Robotics and Automation, 5(3):345-358, 
June 1989. 

[23] P. Viola and W. Wells III. Alignment by maximization of mutual information. In 

International Conference on Computer Vision, pages 16-23, 1995. 
20 [24] A. Zisserman, P. A. Beardsley and L D. Reid. Metric calibration of a stereo rig. In 

Workshop on Representations of Visual Scenes, pages 93-100, 1995. 

[25] P.H.S. Torr and A. Zissennan. Feature based methods for structure and motion 

estimation. In Vision Algorithms Workshop, pages 279-290, Corfu, 1999. 

[26] Z. Zhang, R. Deriche, O. Faugeras, and Q. Luong. A robust technique for matching 
25 two uncalibrated images through the recovery of the unknown epipolar geometry. 

Artificial Intelligence, 78:87-1 19, 1995. 

The disclosures of all publications mentioned in the specification and of the 

publications cited therein are hereby incorporated by reference. 

30 SUMMARY OF THE INVENTION 

This invention* seeks to provide methodologies and systems for combining 
and integrating visual information from various sources. 
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The present invention replaces a requirement of "coherent appearance", 
which is a fundamental assumption in prior art image alignment methods, with a 
requirement of "consistent temporal behavior". The requirement of "consistent temporal 
behavior" is easier to satisfy, for example by producing two sequences of images using 
two cameras moved jointly in space. The present invention is therefore useful not only 
in cases of non-overlapping sequences, but also in cases which are inherently difficult 
for standard image alignment techniques. 

In a preferred embodiment of the present invention the two cameras are 
attached closely to each other, so that their centers of projection are very close, and 
move jointly in space. In this case the induced frame-to-frame transformations within 
each sequence have- correlated behavior across the two sequences. 

A preferred embodiment of the present invention employs correlated 
temporal behavior .to resolve both spatial and temporal transforations between images 
of the two sequences. 

The present invention is therefore, useful for a- variety of real- world 
applications, including: 

(i) Multi-sensor alignment for image fusion. This requires accurate- 
alignment of images obtained by sensors of different sensing modalities, such as 
Infra-Red and visible light. Such images differ significantly in their appearance due to 
different sensor properties. 

(ii) Alignment of image sequences obtained at different zooms. The 
problem here is that different image features are prominent at different image 
resolutions. Alignment of a wide-fleld-of-view sequence with a narrow-field-of-view 
sequence is useful for detecting small zoomed-in objects inside or outside a zoomed-out 
view of the scene. This can be useful in surveillance applications. 

(iii) Generation of wide-screen movies from multiple non-overlapping 
narrow FOV movies, such as in IMAX™ movies. 

It is appreciated that the term "correlated" as used in the specification and 
claims of the present application includes inter alia the following: mathematical 
correlation, statistical correlation and any other co-relationship. 

There is thus provided in accordance with a preferred embodiment of the 
present invention a method for computing at least one of spatial and temporal 
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relationships between at least first and second sequences of representations having 
respective first and second temporal progressions. The method includes employing the 
first and second temporal progressions to obtain at least one of the spatial and temporal 
relationships. 

5 There is also provided in accordance with a preferred embodiment of the 

present invention a method for computing at least one of spatial and temporal 
relationships between at least first and second sequences of representations having 
respective first and second temporal progressions. The method includes employing only 
the first and second temporal progressions to obtain at least one of the spatial and 
10 temporal relationships. 

There is further provided in accordance with a preferred embodiment of the 
present invention a system for computing at least one of spatial and temporal 
relationships between at least first and second sequences of representations having 
respective first and* second temporal progressions. The system includes input 
15 functionality for receiving the first sequence of representations and the first temporal . 
progression and the second sequence of representations and the second temporal 
progression and computation functionality employing at least one of the first sequence 
of representations and the first temporal progression and at least one of the second 
sequence of representations and the second temporal progression to obtain at least one 
20 of the spatial and temporal relationships. 

There is further provided in accordance with another preferred embodiment 
of the present- invention a system for computing at least one of spatial and temporal 
relationships between at least first and second sequences of representations having 
respective first and second temporal progressions. The system includes input 
25 functionality for receiving the first and second temporal progressions and computational 
functionality employing the first and second temporal progressions to obtain at least one 
of the spatial and temporal relationships. 

Further in accordance with a preferred embodiment of the present invention 
the method also includes computing the first and second temporal progressions which 
30 are then employed to obtain at least one of the spatial and temporal relationships. 

Preferably, the representations include visual representations, images and at 
least three-dimensional objects. 



5 



WO 02/47031 



PCT/EL01/01120 



Still further in accordance with a preferred embodiment of the present 
invention the step of employing obtains the spatial relationship, and/or the temporal 
relationship. 

Additionally in accordance with a preferred embodiment of the present 
invention the employing step obtains the spatial and temporal relationships in the 
absence of a spatial overlap between the representations 

Additionally or alternatively, the employing step obtains the spatial and 
temporal relationships in the absence of common spatial information between the 
representations. 

Further in accordance with a preferred embodiment of the present invention 
the step of employing obtains the spatial and temporal relationships in the absence of 
common information between individual ones of the representations belonging to 
different ones of the at least first and second sequences. 

Still further in accordance with a preferred embodiment of the present 
invention the step .of employing obtains the spatial and temporal relationships in the 
absence of common information between any individual ones of the representations 
belonginglo different ones of the at least first and second sequences. 

Additionally in accordance with a preferred embodiment of the present 
invention, the spatial relationship includes at least one parameter of a geometric 
transformation between the at least first and second sequences of representations. 

. Preferably, the spatial relationship includes a 2-dimensional projective 
transformation. 

Alternatively, the spatial relationship includes a 3-dimensional projective 
transformation. 

Further in accordance with a preferred embodiment of the present invention 
the spatial relationship includes a fundamental matrix. 

Still further in accordance with a preferred embodiment of the present 
invention the spatial relationship includes a 2-dimensional parametric transformation. 

Additionally in accordance with a preferred embodiment of the present 
invention the spatial relationship includes an at least 3-dimensional parametric 
transformation. 

Further in accordance with a preferred embodiment of the present invention 
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the spatial relationship includes a 2-dimensional non-parametric transformation. 

Additionally or alternatively, the spatial relationship includes an at least 
3 -dimensional non-parametric transformation. 

Further in accordance with a preferred embodiment of the present invention 
the spatial relationship includes a spatial relationship between individual ones of the 
representations belonging to different ones of the at least first and second sequences. 

Still further in accordance with a preferred embodiment of the present 
invention the spatial relationship includes a spatial relationship between individual ones 
of the representations belonging to different ones of the at" least first and second 
sequences, at least one of the individual ones of the representations being an 
interpolated representation. 

Additionally in accordance with a preferred embodiment of the present 
invention the spatial relationship includes a spatial relationship between individual ones 
of the representations belonging to different ones of the at least first and second 
sequences, one of the individual ones of the representations being an interpolated 
representation and the other of the individual ones of the representations being a 
non-interpolated representation. 

Further in accordance with a preferred embodiment of the present invention 
the temporal relationship includes at least one parameter of a temporal transformation 
between the at least first and second sequences of representations. 

Still further in accordance with a preferred embodiment of the present 
invention the temporal relationship includes a time shift between the at least first and 
second sequences of representations. 

Preferably, the temporal relationship includes an affine transformation in 
time, a parametric transformation in time and/or a non-parametric transformation in 
time. 

Further in accordance* with a preferred embodiment of the present invention 
the temporal relationship includes a temporal relationship in time between individual 
ones of the representations belonging to different ones of the at least first and second 
sequences. 

Still further in accordance with a preferred embodiment of the present 
invention the temporal relationship includes a temporal relationship in time between 
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individual ones of the representations belonging to different ones of the at least first and 
second sequences, at least one of the individual ones of the representations being an 
interpolated representation. 

Additionally in accordance with a preferred embodiment of the present 
invention the temporal relationship includes a temporal relationship in time between 
individual ones of the representations belonging to different ones of the at least first and 
second sequences. Typically, one of the individual ones of the representations is an 
interpolated representation and the other of the individual ones of the representations is 
a non-interpolated representation. 

Preferably, the interpolated representation is interpolated in time and the 
interpolated representation is interpolated in space. 

Further in accordance with a preferred embodiment of the present invention 
the first and second temporal progressions include ordered intra-sequence 
representation-to-representation transformations. 

Still further in accordance with a preferred embodiment of the present 
invention the first and second temporal progressions include ordered intra-sequence 
representation-to-representation transformations resulting from relative motion between 
sensors and a scene. • 

Preferably,.. the intra-sequence representation-to-representation 
transformations include 2-dimensional projective transformations and/or 3-dimensional 
projective transformations. 

Further in accordance with a preferred embodiment of the present invention 
the intra-sequence representation-to-representation transformations include fundamental 
matrices, 2-dimensional parametric transformations, at least 3-dimensional parametric 
transformations, 2-dimensional non-parametric transformations, at least 3-dimensional 
non-parametric transformations and camera matrices. 

Still further in accordance with a preferred embodiment of the present 
invention the step of employing includes correlating the first and second temporal 
progressions. 

Additionally in accordance with a preferred embodiment of the present 
invention the step of employing includes equating properties of the first temporal 
progression and the second temporal progression. 
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Further in accordance with a preferred embodiment of the present invention 
the step of employing includes correlating properties of the first temporal progression 
and the second temporal progression. 

Still further in accordance with a preferred embodiment of the present 
5 invention the step of employing includes equating a sequential application and at least 
one of the intra-sequence representation-to-representation transformations of the first 
temporal progression and an unknown the spatial relationship between the at least first 
and second sequences with a sequential application of the unknown spatial relationship 
between the at least first and second sequences and at least one of the intra-sequence 
1 0 representation-to-representation transformations of the second temporal progression. 

Further in accordance with a preferred embodiment of the present invention 
the step of employing includes equating a composition and at least one of the 
intra-sequence representation-to-representation transformations of the first temporal 
progression and an unknown the spatial relationship between the at least first and 
1 5 second sequences with a composition of the unknown spatial relationship between the at 
least first and second sequences and at least one of the intra-sequence 
representation-to-representation transformations of the second temporal progression. 

Additionally in accordance with a preferred embodiment of the present 
invention the step of employing includes obtaining an unknown the spatial relationship 
20 between the at least first and second sequences by equating a sequential application and 
at least one of the intra-sequence representation-to-representation transformations of the 
first temporal progression and the unknown spatial relationship between the at least first 
and second sequences with a sequential application of the unknown spatial relationship 
between the at least first and second sequences and at least one of the intra-sequence 
25 representation-to-representation transformations of the second temporal progression. 

Additionally in accordance with a preferred embodiment of the present 
invention the step of equating includes equating up to a scale factor. 

Further in accordance with a preferred embodiment of the present invention 
the intra-sequence representation-to-representation transformations include multiple 
30 simple motions taking place at least partially at different times. 

Preferably, the intra-sequence representation-to-representation 
transformations include multiple combinations of multiple simple motions taking place 
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at least partially at different times. 

Preferably, the intra-sequence representation-to-representation 
transformations include multiple complex motions taking place at least partially at 
different times. 

Preferably, the intra-sequence representation-to-representation 
transformations include multiple combinations of multiple complex motions taking 
place at least partially at different times. 

Further in accordance with a preferred embodiment of the present invention 
the first and second temporal progressions include ordered intra-sequence 
representatioh-to-representation transformations at least some of which result from 
relative motion between sensors and a scene. 

Still further in accordance with a preferred embodiment of the present 
invention the step . of employing uses multiple combinations of intra-sequence 
representation-to-representation transformations. 

. Further in accordance with a preferred embodiment of the present invention 
the spatial relationship between the at least first and second sequences of representations 
results from an acquisition relationship, between first and second sensors acquiring 
respective the at least first and second sequences, being fixed over time. 

Still further in accordance with a preferred embodiment of the present 
invention the spatial relationship between the at least first and second sequences of 
representations results from an acquisition relationship, between first and second 
sensors acquiring respective the at least first and second sequences, changes in a known 
way over time. 

Preferably, the acquisition relationship includes relative position, relative 
orientation and relative internal sensor parameters. 

Further in accordance with a preferred embodiment of the present invention 
the acquisition relationship is known. 

Alternatively, the acquisition relationship is not known. 

Additionally in accordance with a preferred embodiment of the present 
invention the at least first and second sequences are acquired generally at the same time. 

Preferably, the at least first and second sequences are acquired generally at 
different times. 
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Further in accordance with a preferred embodiment of the present invention 
the at least first and second sequences represent measurements from the same scene. 

Additionally in accordance with a preferred embodiment of the present 
invention the at least first and second sequences represent measurements from different 
portions of the same scene. 

Additionally or alternatively, the at least first and second sequences 
represent measurements from different overlapping portions of the same scene. 

Further in accordance with a preferred embodiment of the present invention 
the at least first and second sequences represent measurements from different 
non-overlapping portions of the same scene. 

Still further in accordance with a preferred embodiment of the present 
invention the at least first and second sequences represent measurements from different 
scenes. 

Alternatively, the scene is two-dimensional. 
Preferably, the scene is at least three-dimensional. 

Further in accordance with a preferred embodiment of the present invention 
the scene is static. Alternatively, the scene is dynamic. 

Further in accordance with a preferred embodiment of the present invention 
the measurements are generally the same for each sensor. 

Still further in accordance with a preferred embodiment of the present, 
invention the measurements are generally different for each sensor. 

Preferably, the measurements include at least one of illumination, heat, 
radiance, electromagnetic radiation, color, distance, density, sound and speed. 

Further in accordance with a preferred embodiment of the present invention 
the method also includes the step of employing at least one of the spatial and temporal 
relationships for sequence fusion. 

Additionally in accordance with a preferred embodiment of the present 
invention the method also includes step of employing at least one of the spatial and 
temporal relationships for alignment of sequences obtained at different zooms. 

Still further in accordance with a preferred embodiment of the present 
invention the method also includes the step of employing at least one of the spatial and 
temporal relationships for surveillance. 
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Further in accordance with a preferred embodiment of the present invention 
the method also includes the step of employing at least one of the spatial and temporal 
relationships for generation of wide-screen movies from multiple at least partially 
non-overlapping narrow field of view movies. 

Additionally in accordance with a preferred embodiment of the present 
invention the method also includes the step of employing at least one of the spatial and 
temporal relationships for image fusion. 

Further in accordance with a preferred embodiment of the present invention 
. the method also includes the step of employing at least one of the spatial and temporal 
relationships for integrating information contained in the at least first and second 
sequences. 

Still further in accordance with a preferred embodiment of the present 
invention the method also includes the step of employing at least one of the spatial and 
temporal relationships for alignment of images obtained at different zooms. 

Additionally in accordance with- a preferred embodiment of the present' 
invention the method also includes the step of employing at least one of the spatial and 
temporal relationships for comparing information contained in the at least first and 
second sequences. • 

Further in accordance with a preferred embodiment of the present invention 
the method also includes the step of employing at least one of the spatial and temporal 
relationships for finding differences between information contained in the at least first 
and second sequences. 

Still further in accordance with a preferred embodiment of the present 
invention the method also includes the step of employing at least one of the spatial and 
temporal relationships for finding differences between information contained in the at 
least first and second sequences relating to the same scene at different times. 

Additionally in accordance with a preferred embodiment of the present 
invention the method also includes the step of employing at least one of the spatial and 
temporal relationships for integrating information contained in the at least first and 
second sequences and thereby providing an information output which exceeds 
limitations of individual sensors. 

Further in accordance with a preferred embodiment of the present invention 
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the step of employing includes comparing properties of the first temporal progression 
and the second temporal progression. 

Still further in accordance with a preferred embodiment of the present 
invention the computation functionality includes functionality for computing the first 
and second temporal progressions which are then employed to obtain at least one of the 
spatial and temporal relationships. 

Additionally in accordance with a preferred embodiment of the present 
invention the system also includes temporal progression computation functionality for 
computing the first and second temporal progressions and supplying them to the input 
functionality. 

BRIEF DESCRIPTION OF THE DRAWINGS AND APPENDICES 

The present invention will be understood and appreciated more fully from 
the following detailed description, taken in conjunction with the drawings in which: 

Fig. 1 A is a simplified illustration of two cameras, fixed to each other, each 
taking a sequence of images of a portion of a scene, the portions of the scene 
photographed by the two cameras being non-overlapping; 

Fig. IB illustrates inherent ambiguities in the relative spatial relationship of 
images in the sequences taken as shown in Fig. 1 A; 

Fig. 1C illustrates portions of two sequences of images taken as shown. in 
Fig. 1 A and the unknown temporal relationship between the sequences; 

Fig. ID illustrates pairs of images forming part of the sequences of images 
taken as shown in Fig. 1A, whose spatial and temporal relationships are determined in 
accordance with the present invention; 

Fig. 2A is a simplified illustration of two cameras, fixed to each other, each 
taking a sequence of images of a portion of a scene, the portions of the scene being 
photographed at significantly different zooms by the two cameras; 

Fig. 2B illustrates inherent ambiguities in the relative spatial relationship of 
images in the sequences taken as shown in Fig. 2A; 

Fig. 2C illustrates portions of two sequences of images taken as shown in 
Fig. 2A and the unknown temporal relationship between the sequences; 

Fig. 2D illustrates pairs of images forming part of the sequences of images 
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taken as shown in Fig. 2A, whose spatial and temporal relationships are determined in 
accordance with the present invention; 

Fig. 3A is a simplified illustration of two sensors, fixed to each other, each 
taking a sequence of images of a portion of a scene, the portions of the scene being 
imaged by the two sensors employing different sensing modalities; 

Fig. 3B illustrates inherent ambiguities in the relative spatial relationship of 
images in the sequences taken as shown in Fig. 3A; 

Fig. 3C illustrates portions of two sequences of images taken as shown in 
Fig. 3 A and the unknown temporal relationship between the sequences; 

Fig. 3D illustrates pairs of images forming part of the sequences of images 
taken, as shown in Fig. 3 A, whose spatial and temporal relationships are determined in 
accordance with the present invention; 

Fig. 4A is a simplified illustration of a scene being photographed at two 
different times, producing two corresponding sequences of images, the path traveled by 
a camera used for photographing being generally identical for both sequences; 

Fig. 4B illustrates inherent ambiguities in the relative spatial relationship of 
images in the sequences taken as shown in Fig. 4A; 

Fig. 4C illustrates portions of two sequences of images taken as shown in 
Fig. 4A and the unknown temporal relationship between the sequences; 

Fig. 4D illustrates pairs of images forming part of the sequences of images 
taken as shown in Fig. 4A, whose spatial and temporal relationships are determined in ' 
accordance with the present invention; 

Fig. 5 A illustrates the relationships between image to image transformations 
within two sequences, induced by motion of two cameras along an axis, the two 
cameras being arranged at 180 degrees with respect to each other along the axis; 

Fig. 5B illustrates the relationships between image to image transformations 
within two sequences, induced by motion of two cameras along an axis, the two 
cameras being arranged at 90 degrees with respect to each other, one of the cameras 
being aligned along the axis of motion; 

Fig. 5C illustrates the relationships between image to image transformations 
within two sequences, induced by motion of two cameras along an axis, the two 
cameras being directed in the same direction perpendicular to the direction of motion 
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but at different zooms; 

Figs. 6A, 6B and 6C together illustrate that employing one type of 
transformation may not be sufficient for resolving ambiguities in the spatial and 
temporal relationships between sequences but that employing multiple different types of 
5 transformations reduces ambiguities in the spatial and temporal relationships between 
sequences; 

Fig. 7 is a simplified illustration of a complex motion of two cameras, fixed 
to each other, each taking a sequence of images of a portion of a scene, the portions of 
the scene photographed by the two cameras being non-overlapping; 
1 0 Fig. 8 is a simplified illustration portions of two sequences of images taken 

as shown in Fig. 7, wherein the two sequences are spatially related by a fixed and 
unknown inter-camera homography and temporally related by a fixed and unknown 
time shift. 

Fig. 9 is a simplified functional block diagram of a preferred process of 
1 5 creating two sequences of images and aligning them. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
Reference is now made to Figs. 1A - ID which illustrate resolution of 
spatial and temporal relationships between sequences of images taken by two moving 
20 cameras fixed to each other in accordance with the present invention. 

As seen in Fig. 1A, two cameras, designated respectively by reference 
numerals 100 and 102 are fixed to each other in any suitable manner. Each camera takes 
a sequence of images of a portion of a scene as the cameras move while they are fixed 
together. The movement of the cameras may be any suitable movement, such as rotation 
25 and/or translation in one or more dimensions and relative to any suitable point. Thus, for 
example, the two cameras 100 and 102 may rotate about the optical axis of one of the 
cameras or about any other axis. Similarly, translation of the cameras may occur in any 
suitable direction. 

In the present case (Figs. 1A - ID) portions of the scene photographed by 
30 the two cameras are non-overlapping and are designated respectively by reference 
numerals 104 and 106. 

Turning now to Fig. IB, it is seen that the portions 104 and 106 of the scene 
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of Fig. 1A may be represented by corresponding images 108 and 110. As seen in Fig. 
1C, images 108 and 110 each belong to a sequence of images, each produced by one of 
the moving cameras 100 and 102, the respective sequences being designated by 
reference numerals 1 12 and 1 14. It is seen that sequence 1 12 also includes images 122, 
124 and 126. Likewise, sequence 1 14 also includes images 130, 132 and 134. 

A problem addressed by the present invention is that the visual- information 
contained in individual pairs of images, one belonging to sequence 112 and the other 
belonging to sequence 114, (e.g. (108, 110), (122, 130) or (122, 134)), is sufficient to 
establish neither the spatial nor the temporal relationship between two images of a pair. 
More generally, the; visual information contained in individual pairs of images, one of 
the pair, belonging to sequence 1 12 and the other of the pair belonging to sequence 1 14, 
is sufficient to establish neither the spatial nor the temporal relationship between the 
two sequences 1 12 and 1 14. 

In the context of Figs. 1A - ID, two images are the to be in the same 
"temporal relationship 1 ' if they are taken at the same time. 

The unknown spatial relationship of images 108 and 1 10 is seen graphically 
by considering three examples of possible relative spatial relationships shown in Fig. IB 
and designated by reference numerals 116, 118 and 120. In each example, the two 
images 108 and 110 are placed, in a different spatial relationship, all of which are 
consistent with the visual content of the images 108 and 110. 

The unknown temporal relationship of sequences 112 and 114 is seen 
graphically by considering Fig. 1C. It is appreciated that it is not apparent from a 
cursory examination of sequences 1 12 and 1 14, which images in sequence 1 12 are taken 
at the same time as which images in sequence 114. 

The present invention provides a system and technique for determining the 
correct relationship between the images 108 and 110 and more generally the correct 
spatial and temporal relationships between sequences 1 12 and 1 14, as shown in Fig. ID. 
Thus, it may be appreciated that the present invention determines which image in 
sequence 112 corresponds in time with which image in sequence 114 and further 
determines the spatial relationship between the images which correspond in time. 

Thus it is seen in Fig. ID that image A (108) in sequence 112 is found to 
correspond in time with image b (1 10) in sequence 1 14. The correct spatial relationship 
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between images A (122) and b (110) is shown in Fig. ID at reference numeral 150. 
Similarly, image B (122) in sequence 112 is found to correspond in time with image c 
(132) in sequence 114 and image C (124) in sequence 112 is found to correspond in 
time with image d (134) in sequence 114. The correct spatial relationship between 
images B (122) and c (132) is shown in Fig. ID at reference numeral 152 and the 
correct spatial relationship between images C (124) and d (134) is shown in Fig. ID at 
reference numeral 154. 

The present invention employs an appreciation that despite there being no 
common static visual information in the two sequences, the image to image dynamics in 
each of the two sequences are nevertheless related both spatially and temporally. The 
present invention utilizes the relationship between the dynamics to correlate the two 
sequences in time and space. 

. As will be described hereinbelow in detail, the image to image dynamics 
preferably are expressed by spatial transformations over time between images in each 
sequence. The problem of aligning sequences of images in time and space is thus 
reduced to a problem of determining the relationship between sequences of such 
transformations in time and in space. 

It is appreciated from a consideration of Fig. ID, that by employing the 
present invention aligned sequences may be realized, which sequences may then be 
utilized to localize two portions of a scene with respect to each other, despite the 
absence of spatial overlap therebetween. 

Reference is now made to Figs. 2A - 2D which illustrate the resolution of 
spatial and temporal relationships between sequences of images taken by two moving 
cameras fixed to each other in accordance with the present invention, the portions of the 
scene being photographed at significantly different zooms by the two cameras. 

As seen in Fig. 2A, two cameras, designated respectively by reference 
numerals 200 and 202 are fixed to each other in any suitable manner. Each camera takes 
a sequence of images of a portion of a scene as the cameras move while they are fixed 
together. The movement of the cameras may be any suitable movement, such as rotation 
and/or translation in one or more dimensions and relative to any suitable point. Thus, for 
example, the two cameras may rotate about the optical axis of one of the cameras or 
about any other axis. Similarly, translation of the cameras may occur in any suitable 
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direction. 

In the present case (Figs. 2A - 2D) portions of the scene photographed by 
the two cameras are imaged at significantly different zooms, thus imaging various 
features in the scene at different spatial scales. Two portions of the scene, which need 
not necessarily be overlapping, are designated respectively by reference numerals 204 
and 206. 

Turning now to Fig. 2B, it is seen that the portions 204 and 206 of the scene 
of Fig. 2 A may be represented by corresponding images 208 and 210, respectively. As 
seen in Fig. 2C, images 208 and 210 each belong to a sequence of images, each 
produced by one of the moving cameras 200 and 202, the respective sequences being 
designated by reference numerals 212 and 214. It is seen that sequence 212 also 
includes images 222, 224 and 226. Likewise, sequence 214 also includes images 230, 
232 and 234. 

It is appreciated that features imaged in the zoomed-in image 210 may be 
different from the features imaged in the zoomed-out image 208 due to the different 
spatial scales employed by the cameras. Thus, for example, individual leaves of a tree 
may be discerned in image 210, while such leaves may not be discernible in image 208. 
Similarly trees may be discerned in image 208, while only leaves and branches are 
discernible in image 210. 

A problem addressed by the present invention is that the visual information 
contained in individual pairs of images, one belonging to sequence 212 and the other 
belonging to sequence 214, (e.g. (208, 210), (222, 230) or.(222, 234)), is sufficient to 
establish neither the spatial nor the temporal relationship between two images of a pair. 
More generally, the visual information contained in individual pairs of images, one of 
the pair belonging to sequence 212 and the other of the pair belonging to sequence 214, 
is sufficient to establish neither the spatial nor the temporal relationship between the 
two sequences 2 1 2 and 214. 

In the context of Figs. 2A - 2D, two images are the to be in the same 
"temporal relationship" if they are taken at the same time. 

The unknown spatial relationship of images 208 and 210 is seen graphically 
by considering three examples of possible relative spatial relationships shown in Fig. 2B 
and designated by reference numerals 216, 218 and 220. In each example, the two 

18 



WO 02/47031 



PO7DL01/01120 



images 208 and 210 are placed in a different spatial relationship with each other, all of 
which are consistent with the visual content of the images 208 and 210. 

The unknown temporal relationship of sequences 212 and 214 is seen 
graphically by considering Fig. 2C. It is appreciated that it is not apparent from a 
cursory examination of sequences 212 and 214, which images in sequence 212 are taken 
at the same time as which images in sequence 214. 

The present invention provides a system and technique for determining the 
correct relationship between the images 208 and 210 and more generally the correct 
spatial and temporal relationships between sequences 212 and 214, as shown in Fig. 2D. 
Thus, it may be appreciated that the present invention determines which image in 
sequence 212 corresponds in time with which image in sequence 214 and further 
determines the spatial relationship between the images which correspond in time. 

Thus it is seen in Fig. 2D that image A (208) in sequence 212 is found to 
correspond in time with image b (210) in sequence 214. The correct spatial relationship 
between images A (208) and b (210) is shown in Fig. 2D at reference numeral 250. 
Similarly, image B (222) in sequence 212 is found to correspond in time with image c 
(232) in sequence 214 and image C (224) in sequence 212 is found to correspond in " 
time with image d (234) in sequence 214. The correct spatial relationship between 
images B (222) and c (232) is shown in Fig. 2D at reference numeral 252 and the 
correct spatial relationship between images C (224) and d (234) is shown in Fig. 2D at 
reference numeral 254. 

The present invention employs an appreciation that despite there not 
necessarily being any common static visual information in the two sequences, the image 
to image dynamics in each of the two sequences are nevertheless related both spatially 
and temporally. The present invention utilizes the relationship between the dynamics to 
correlate the two sequences in time and space. 

As will be described hereinbelow in detail, the image to image dynamics 
preferably are expressed by spatial transformations over time between images in each 
sequence. The problem of aligning sequences of images in time and space is thus 
reduced to a problem of determining the relationship between sequences of such 
transformations in time and in space. 

It is appreciated from a consideration of Fig. 2D, that by employing the 
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present invention aligned sequences may be realized, which sequences may then be 
utilized to provide high resolution details of a portion of a scene, localized with respect 
to a zoomed-out view of the scene, providing a wide field of view context. 

Reference is now made to Figs. 3A - 3D which illustrate resolution of 
spatial and temporal relationships between sequences of images taken by two moving 
sensors fixed to each other in accordance with the present invention, the portions of the 
scene being imaged by the two sensors employing different sensing modalities. 

As seen in Fig. 3A, two sensors, such as an infra red (IR) camera 300 and a 
visible light (VL) camera 302, are fixed to each other in any suitable manner. Each 
sensor takes a sequence of images of a portion of a scene as the sensors move while 
they are fixed together. The movement of the sensors may be any suitable movement, 
such as rotation and/or translation in one or more dimensions and relative to any 
suitable point. Thus, for example, the two sensors may rotate about the optical axis of 
one of the sensors or about any other axis. Similarly, translation of the sensors may 
occur in any suitable. direction. 

In the present case (Figs. 3 A - 3D) portions of die scene sensed by the two 
sensors are imaged at different wavelength ranges, thus imaging various features in the 
scene in different modalities. Two portions of the scene, which need not necessarily be 
overlapping or at the same zoom, are designated by reference numerals 304 and 306. 

Turning now to Fig. 3B, it is seen that the portions 304 and 306 of the scene 
of Fig. 3 A may be represented by corresponding images 308 and 310. As seen in Fig. 
3C, images 308 and 3 10 each belong to a sequence of images, each produced by one of 
the moving sensors 300 and 302, the respective sequences being designated by 
reference numerals 312 and 314. It is seen that sequence 3 12 also includes images 322, 
324 and 326. Likewise, sequence 314 also includes images 330, 332 and 334. 

It is appreciated that features imaged in the infra-red (IR) image 308 may be 
different from the features imaged in the visible light (VL) image 310 due to the 
different sensing modalities employed by the sensors. Thus, for example, hot regions 
may be discerned in image 308, while heat is not sensed in image 310. Similarly the 
visual appearance of a bus may be discerned in image 310, while features appearing 
only in visible light are not captured in image 308. 

A problem addressed by the present invention is that the visual information 
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contained in individual pairs of images, one belonging to sequence 312 and the other 
belonging to sequence 314, (e.g. (308, 310), (322, 330) or (322, 334)), is sufficient to 
establish neither the spatial nor the temporal relationship between two images of a pair. 
More generally, the visual information contained in individual pairs of images, one of 
5 the pair belonging to sequence 312 and the other of the pair belonging to sequence 314, 
is sufficient to establish neither the spatial nor the temporal relationship between the 
two sequences 312 and 314. 

In the context of Figs. 3A - 3D, two images are the to be in the same 
"temporal relationship" if they are taken at the same time. 

10 The unknown spatial relationship of images 308 and 310 is seen graphically 

by considering three examples of possible relative spatial relationships shown in Fig. 3B 
and designated by reference numerals 316, 318 and 320. In each example, the two 
images 308 and 310 are placed in a different spatial relationship, all of which are 
consistent with the visual content of the images 308 and 310. 

15 The unknown temporal relationship of sequences 312 and 314 is seen 

graphically by considering Fig. 3C. It is appreciated . that it is not apparent from a 
cursory examination of sequences 312 and 314, which images in sequence 312 are taken 
at the same time as which images in sequence 314, 

The present invention provides a system and technique for determining the 

20 correct relationship between the images 308 and 310 and more generally the correct 
spatial and temporal relationships between sequences 312 and 314, as shown in Fig. 3D. 
Thus, it may be appreciated that the present invention determines which image in 
sequence 312 corresponds in time with which image in sequence 314 and further 
determines the spatial relationship between the images which correspond in time. 

25 Thus it is seen in Fig. 3D that image A (310) in sequence 314 is found to 

correspond in time with image a (308) in sequence 312. The correct spatial relationship 
between images A (310) and a (308) is shown in Fig. 3D at reference numeral 350. 
Similarly, image B (330) in sequence 314 is found to correspond in time with image b 
(322) in sequence 312, image C (332) in sequence 314 is found to correspond in time 

30 with image c (324) in sequence 312 and image D (334) in sequence 314 is found to 
correspond in time with image d (326) in sequence 312. The correct spatial relationship 
between images B (330) and b (322) is shown in Fig. 3D at reference numeral 352, the 
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correct spatial relationship between images C (332) and c (324) is shown in Fig. 3D at 
reference numeral 354 and the correct spatial relationship between images D (334) and 
d (326) is shown in Fig. 3D at reference numeral 356. Thus it may be realized that the 
hot regions captured in the IR image sequence 312 are in fact the radiator and the 
exhaust gases of the bus. 

The present invention employs an appreciation that despite there not 
necessarily being any common static visual information in the two sequences, the image 
to image dynamics in each of the two sequences are nevertheless related both spatially 
and temporally. The present invention utilizes the relationship between the dynamics to 
correlate the two sequences in time and space, without requiring understanding or 
interpretation of the features captured by the sequences. 

As will be described hereinbelow in detail* the. image to image dynamics 
preferably are expressed by spatial transformations over time between images in each 
sequence. The problem of aligning sequences of images in time and space is thus 
reduced to a problem of determining the relationship between sequences of such 
transformations in time and in space. 

It is appreciated from a consideration of Fig. 3D, that by employing the 
present invention aligned sequences may be realized, which sequences may then be 
integrated to provide a composite sequence displaying the totality of information 
captured by both sequences. In this example, .the composite sequence thus captures 
information over a wavelength range which is beyond that of the individual sensors 300 
and 302. 

Reference is now made to Figs. 4A - 4D which illustrate resolution of 
spatial and temporal relationships between sequences of images of a scene taken at two 
different times, producing two corresponding sequences of images. A camera used for 
producing the sequences of images is fixed to a element which travels along a path that 
is generally identical for both sequences. 

As seen in Fig. 4A, a camera mounted onto an element, traveling along a 
generally identical path, such as a railroad track, images a scene at two different times, 
here typically summer and winter. The camera used in the summer is designated by 
reference numeral 400, while the camera used in the winter is designated by reference 
numeral 402, it being appreciated that the same camera or different cameras may be 
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employed. The cameras may have the same or different settings. 

Each camera takes a sequence of images of the scene. In this case the scene 
being imaged at the two different times, has two different appearances, as designated by 
reference numerals 404 and 406. Reference numeral 404 designates a view of a train 

5 bearing camera 400, traveling along railroad track 407 in the summer. Reference 
numeral 406 designates a view* of a train bearing camera 402 traveling along railroad 
track 407 in the winter. 

Turning now to Fig. 4B, it is seen that cameras 400 and 402 image portions 
of the views designated by reference numerals 404 and 406 of Fig. 4A, generating 

10 respective images 408 and 410. As seen in Fig. 4C, images 408 and 410 each belong to 
a sequence of images, each produced by one of the moving cameras 400 and 402, the 
respective sequences being designated by reference numerals 412 and 414. It is seen 
that sequence 412 also includes images 422, 424 and 426. Likewise, sequence 414 also 
includes images 430, 432 and 434. It is appreciated that the images in sequence 412 are 

15 not necessarily spaced in time (taken at the same .time differences) as the images in 
sequence 414. 

It is appreciated that features imaged in the summer may be different from 
the features imaged in the winter due, for example to snow cover. Thus, for example, a 
house seen in sequence 412 is not visible in sequence 414, because it is covered by 

20 snow. Similarly snow seen in sequence 414 is not visible in sequence 412. 

A problem addressed by the present invention is that the visual information 
contained in individual pairs of images, one belonging to sequence 412 and the other 
belonging to sequence 414, (e.g. (408, 410), (422, 430) or (422, 434)), is sufficient to 
establish neither the spatial nor the temporal relationship between two images of a pair. 

25 More generally, the visual information contained in individual pairs of images, one of 
the pair, belonging to sequence 412 and the other of the pair belonging to sequence 414, 
is sufficient to establish neither the spatial nor the temporal relationship between the 
two sequences 412 and 414. 

In the context of Figs. 4A - 4D, two images are the to be in the same 

30 "temporal relationship" if an element to which the camera is fixed is at the same 
location along the path. It is appreciated that the time differences between images 422, 
424, 408 and 426 in sequence 412 may or may not be the same as the time differences 
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between the images 430, 432, 410 and 434 in sequence 414. 

The unknown spatial relationship of images 408 and 410 is seen graphically 
by considering three examples of possible relative spatial relationships shown in Fig. 4B 
and designated by reference numerals 416, 418 and 420. In each example, the two 
images 408 and 410 are placed in a different spatial relationship, all of which are 
consistent with the visual content of the images 408 and 410. 

The unknown temporal relationship of sequences 412 and 414 is seen 
graphically by considering Fig. 4C. It is appreciated that it is not apparent from a 
cursory examination of sequences 412 and 414, which images in sequence 412 are taken 
from the same position along the railroad track 407 as images in sequences 414. 

The present invention provides a system and technique for determining the 
correct relationship between the images 408. and 410 and more generally the correct 
spatial and temporal relationships between sequences 412 and 414, as shown in Fig. 4D. 
Thus, it may be appreciated that the present invention determines . which image in 
sequence 412 corresponds temporally with which image in sequence 414 and further 
determines the spatial relationship between the images which correspond temporally. 

Thus it is seen in Fig. 4D that image A (422) in sequence 412 is found to 
correspond temporally with image a (430). in sequence 414. The correct spatial 
relationship between images A (422) and a (430) is shown in Fig. 4D at reference * 
numeral 450. Similarly, image B (424) in sequence 412 is found to correspond 
temporally with image b (432) in sequence 414, image C (408) In sequence 412 is found 
to correspond temporally with image c (410) in sequence 414 and image D (426) in 
sequence 412 is found to correspond temporally with image d (434) in sequence 414. 
The correct spatial relationship between images B (424) and b (432) is shown in Fig. 4D 
at reference numeral 452, the correct spatial relationship between images C (408) and c 
(410) is shown in Fig. 4D at reference numeral 454 and the correct spatial relationship 
between images D (426) and d (434) is shown in Fig. 4D at reference numeral 456. 
Thus the location of the house buried under the snow may be determined in sequence 
414 although the house is -not visible in that sequence. 

The present invention employs an appreciation that despite there not 
necessarily being any common static visual information in the two sequences, the image 
to image dynamics in each of the two sequences are nevertheless related both spatially 
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and temporally. The present invention utilizes the relationship between the dynamics to 
correlate the two sequences spatially and temporally, without requiring understanding or 
interpretation of the features captured by the sequences. 

As will be described hereinbelow in detail, the image to image dynamics 
preferably are expressed by spatial transformations over time between images in each 
sequence. The problem of aligning sequences of images spatially and temporally is thus 
reduced to a problem of determining the relationship between sequences of such 
transformations. 

It is appreciated from a consideration of Fig. 4D, that by employing the 
present invention aligned sequences may be realized, which sequences may then be 
employed to determine the location of objects visible in only some sequences across 
other sequences. The aligned sequences may also be employed to detect changes in a 
scene, such as man-made changes, which take place during a time interval between 
acquisitions of different sequences. . 

It is also appreciated that the spatial relationship may also be produced by 
moving at least one camera at different times and/or in different locations, along 
generally different trajectories, wherein the at least two trajectories are correlated as if 
. they were produced by two cameras that are mounted rigidly on the same device or rig, 
and moved jointly in space. 

It is appreciated that Figs. 4A - 4D show one example of a possible 
visualization of the output of the method of the present invention. However, the method 
is not limited to this particular output or this particular visualization. It is appreciated 
that, for example, the output can be numerical, in the form of spatial and temporal 
transformations, or the output can be visual in the form of aligned and/or integrated 
video sequences. 

When numerical outputs are provided, the output transformations can be 
provided in various possible coordinate systems and when visual outputs are provided, 
there are many possible visualizations. 

Reference is now made to Figs. 5A - 5C, which illustrate an important 
inventive principle underlying the present invention. As noted hereinabove with 
reference to Figs. 1A - 4D, a difficulty exists in establishing spatial and temporal 
correspondence between images that have little or no visual information in common. 
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Noting that each of the images forms part of a sequence of images which 
have substantial visual information in common, the present invention overcomes this 
difficulty by considering intra-sequence spatial transformations between images in each 
sequence and by then correlating series of such intra-sequence spatial transformations 
across the sequences. 

It is appreciated that while visual information common . to images within a 
sequence is used to compute the intra-sequence spatial transformations, these 
transformations, once computed, do not include any visual information and can 
therefore be readily correlated across sequences. Thus, once series of intra-sequence 
transformation are found, there is no longer any need to refer back to the visual 
information in the images themselves. 

It is a particular feature of the present invention that the series of 
intra-sequence spatial transformations of images taken by cameras whose motion is 
spatially and temporally interrelated, such as described in any of Figs. 1A - 4D, are 
spatially and temporally correlated. Figs. 5A T 5C and the following description present 
examples of these correlations which can be employed in accordance with the present 
invention to derive the spatial and temporal relationship, notwithstanding a lack of 
common visual information across the image sequences. 

Specific reference is now made to Fig. 5A, which illustrates the 
relationships between image to image transformations within two sequences, induced by 
motion of first and second cameras 500 and 502 along an axis 504, the two cameras 
being arranged at 180 degrees with respect to each other along the axis. Camera 500 
captures a sequence of images 506, while camera 502 captures a sequence of images 
508. 

Sequence 506 includes inter alia images 510, 512 and 514, while sequence 
508 includes inter alia images 520, 522 and 524. It is seen that the motion of camera 
500 along axis 504 in a direction indicated by arrow 526 causes an imaged scene 
including a house to appear smaller in image 512 than in image 510. This relationship is 
represented by an intra-sequence image-to-image spatial transformation and the 
graphical illustration of the transformation from image 510 to image 512 is designated 
by reference numeral 530. The arrows shown in the transformation represent the 
displacement of the corresponding points of the image 512 as the image 512 is 
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transformed to image 5 1 0. Similarly, it is seen that the motion of camera 500 along axis 
504 in a direction indicated by arrow 526 causes the imaged scene including the house 
to appear smaller in image 514 than in image 512. This relationship is represented by an 
intra-sequence image-to-image spatial transformation designated by reference numeral 
5 532. 

It is also seen that the motion of camera 502 along axis 504 in a direction 
indicated by arrow 526 causes an imaged scene including a tree to appear larger in 
image 522 than in image 520. This relationship is represented by an intra-sequence 
image-to-image spatial transformation designated by reference numeral 540. Similarly, 

10 it is seen that the motion of camera 502 along axis 504 in a direction indicated by arrow 
526 causes the imaged scene including the tree to appear larger in image 524 than in 
image 522. This relationship is represented by an intra-sequence image-to-image spatial 
transformation designated by reference numeral 542. 

It is appreciated from a comparison of the series of transformations 530 and 

15 532 with the series of transformations 540 and 542 that cameras 500 and 502 appear to 
be directed in opposite directions as they move jointly. 

Specific reference is now made to Fig. 5B, which* illustrates the 
relationships between image to image transformations within two sequences, induced by 
motion of first and second cameras 550 and 552 along an axis 554, the two cameras 

20 being arranged at 90 degrees with respect to each other, one of the cameras being 
aligned substantially parallel to the axis of motion 554. Camera 550 captures a sequence 
of images 556, while camera 552 captures a sequence of images 558. 

Sequence 556 includes inter alia images 560, 562 and 564, while sequence. 
558 includes inter alia images 570, 572 and 574. It is seen that the motion of camera 

25 550 along axis 554 in a direction indicated by arrow 576 causes an imaged scene 
including a house to appear shifted sideways to the left in image 562 as compared with 
image 560. This relationship is represented by an intra-sequence image-to-image spatial, 
transformation designated by reference numeral 580. Similarly, it is seen that the motion 
of camera 550 along axis 554 in a direction indicated by arrow 576 causes the imaged 

30 scene including the house to appear shifted sideways to the left in image 564 as 
compared with image 562. This relationship is represented by an intra-sequence 
image-to-image spatial transformation designated by reference numeral 582. 
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It is also seen that the motion of camera 552 along axis 554 in a direction 
indicated by arrow 576 causes an imaged scene including a tree to appear larger in 
image 572 than in image 570. This relationship is represented by an intra-sequence 
image-to-image spatial transformation designated by reference numeral 590. Similarly, 
5 it is seen that the motion of camera 552 along axis 554 in a direction indicated by arrow 
576 causes the imaged scene including the tree to appear larger in image 574 than in 
image 572. This relationship is represented by an intra-sequence image-to-image spatial 
transformation designated by reference numeral 592. 

It is appreciated from a comparison of the series of transformations 580 and 
0 582 with the series of transformations 590 and 592 that cameras 550 and 552 appear to 
be directed perpendicularly to each other as they move jointly. 

Specific reference is now made to Fig. 5C, which illustrates the. 
relationships between image to image transformations within two sequences, induced by 
motion of first and second cameras 600 and 602 along an axis 604, the two cameras 
being directed in the same direction perpendicular to the direction of motion along axis 
604 but at different zooms. Camera 600 captures a sequence of images 606, while 
camera 602 captures a sequence of images 608. 

Sequence 606 includes inter alia images 610, 612 and 614, while sequence 
608 includes inter alia images 620, 622 and 624. It is seen that the motion of camera 
600 along axis 604 in a direction indicated by arrow 626 causes an imaged scene 
including a house to appear shifted sideways to the left in image 612 as compared with 
image 610. This relationship is represented by an intra-sequence image-to-image spatial 
transformation designated by reference numeral 630. Similarly, it is seen that the motion 
of camera 600 along axis 604 in a direction indicated by arrow 626 causes the imaged 
scene including the house to appear shifted sideways to the left in image 614 as 
compared with image 612. This relationship is represented by an intra-sequence 
image-to-image spatial transformation designated by reference numeral 632. 

It is also seen that the motion of camera 602 along axis 604 in a direction 
indicated by arrow 626 causes an imaged scene including the house to appear shifted 
sideways to the left in image 622 as compared with image 620. This relationship is 
represented by an intra-sequence image-to-image spatial transformation designated by 
reference numeral 640. Similarly, it is seen that the motion of camera 602 along axis 
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604 in a direction indicated by arrow 626 causes the imaged scene including the house 
to appear shifted sideways to the left in image 624 as compared with image 622. This 
relationship is represented by an intra-sequence image-to-image spatial transformation 
designated by reference numeral 642. It is noted that the sideways shifts represented by 
transformations 640 and 642 are larger than those represented by transformations 630 
and 632 due to the difference in zooms. 

It is appreciated from a comparison of the series of transformations 630 and 
632 with the series of transformations 640 and 642 that cameras 600 and 602 appear 
directed in the same direction perpendicular to the direction of motion along axis 604 
but at different zooms as they move jointly. 

• Reference is now made to Figs. 6 A, 6B and 6C, which together illustrate 
that computational functionality employing one type of transformation may not ' be 
sufficient for resolving ambiguities in the spatial and temporal relationships between 
sequences but that employing multiple different types of transformations reduces 
ambiguities in the spatial and temporal relationships between sequences. 

Turning to Fig. 6 A, it is seen that intra-sequence spatial transformations 650 
and 652 representing two oppositely directed sideways movements, each belonging to a 
sequence taken by a different camera, can represent at least two different arrangements 
of the cameras. One such arrangement, designated by reference numeral 654 is of first 
and second cameras, 656 and 658 respectively, moving in a direction 660 along an axis 
662 and being arranged in a mutual upside down arrangement and both directed in 
parallel perpendicular to axis 662, 

Another possible arrangement, designated by reference numeral 664, is of 
cameras 656 and 658 being arranged at 180 degrees with respect to each other, directed 
perpendicular to the axis 662 and moving along axis 662 in a direction 660 therealong. 

Turning to Fig. 6B, it is seen that intra-sequence spatial transformations 670 
and 672 representing forward and backward motion respective, each belonging to a 
sequence taken by a different camera, can represent at least two different arrangements 
of the cameras. One such arrangement, designated by reference numeral 664*is of first 
and second cameras, 656 and 658 respectively, moving in a direction 680 along an axis 
682 and being arranged at 180 degrees with respect to each other along axis 682. 

Another possible arrangement, designated by reference numeral 684, is of 
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cameras 656 and 658 being arranged at 180 degrees with respect to each other along 
axis 682 for motion in a direction 680 therealong, wherein one of the cameras 656 is 
rotated about axis 682 with respect to the other one of the cameras 658. 

Turning to Fig. 6C, it is seen that two series of intra-sequence spatial 
5 transformations 690 and 692, each taken by a different moving camera, and containing 
transformations of different types, are employed by computational functionality to 
resolve ambiguities of the type described hereinabove with reference to Figs. 6A and 
6B. Series 690 comprises transformation 650 (Fig. 6 A) followed by transformation 670 
(Fig. 6B), while' series 692 comprises transformation 652 (Fig. 6A) followed by 
1 0 transformation 672 (Fig. 6B). 

It is seen that only one of the three camera configurations shown in Figs. 
6A, 6B and 6C is possible, namely that configuration designated by reference numeral 
664 in Figs. 6A and 6B. 

Figs. 1A to 4D described hereinabove present four typical applications of 
15 the present invention. In these four applications the present invention provides 
alignment between images of two sequences of images notwithstanding that there is 
insufficient similarity between any image of one sequence to any image of the other 
sequence to enable the prior art to provide image to image alignment. "Similarity" of 
images is used here in a broad sense and includes, inter alia, gray-level similarity, 
20 feature similarity, similarity of frequencies and statistical similarity, such as mutual 
information. Consequently, in the prior art, at least partial overlap between the images is 
required to obtain similarity. 

In contrast to the prior art, which requires overlap between images in order 
to mutually align them, the present invention employs correlated temporal behavior 
25 between the two sequences, as shown and described with reference to Figs. 5A to 6C, to 
resolve both the spatial and the temporal transformations between images of the two 
sequences. A features of the present invention is the ability to replace the 

requirement for ''coherent appearance", which is a fundamental assumption in the prior 
art, with the requirement for "consistent temporal behavior" which is often easier to 
30 satisfy, for example by moving the two cameras jointly in space. 

In one application of the present invention, shown and described 
hereinabove with reference to Figs. 1A to ID, the two sequences are produced 
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concurrently by two cameras moving jointly in space. The term 'concurrently' here 
designates that the two sequences are produced at least partially during the same period 
of time. The concurrency does not require that images, or frames, of the two sequences 
be produced simultaneously or even at the same rate. 'Moving jointly' here means that 
the two cameras have, for practical purposes, the same center of projection throughout 
the sequences. This may be realized, for example by fixedly attaching both cameras to 
the same moving rig. In reality, the centers of projection of the two cameras are not at 
precisely the same location. However in the mathematical treatment which follows, the 
centers of projection of both cameras are considered as if they were at the same 
location. It is further appreciated that the mathematical principles employed in the 
mathematical, treatment are also applicable to situations where the centers of projection 
of the cameras are clearly not at the same location but are at locations whose spatial 
relationship is fixed. It is additionally appreciated that the mathematical principles 
employed in the mathematical treatment are also applicable to situations where the 
centers of projection of the cameras are clearly not at the same location but are at 
locations having an at least partially known spatial/temporal relationship. 

In another application of the present invention, shown and described 
hereinabove with reference to Figs. 2A to 2D and Figs. 3A to 3D, the two sequences 
contain images that are partially overlapping. However the similarity between the 
images is very low or even nonexistent, namely, no detail of any image, of one sequence 
can be identified in any image of the second sequence. 

In the applications shown and described hereinabove with reference to Figs. 
2A to 2D, the two sequences are produced by cameras employing significantly different 
zoom. Similar applications may relate to sequences produced by cameras having 
significantly different magnification or resolution. 

In the applications shown and described hereinabove with reference to Figs. 
3 A to 3D, the two sequences are produced by cameras employing different sensing 
modalities. Typical sensing modalities include: using a photographic camera, such as a 
still film camera, a digital camera, a video camera or a motion film camera; employing 
infra-red imaging equipment, radar, x-ray, CAT-scan, mri or other electromagnetic 
imaging equipment, acoustic imaging equipment, such as ultrasound, sonar and 
sub-acoustic geophysical surveillance equipment, satellite remote sensing equipment. 
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In yet another application of the present invention shown and described 
hereinabove with reference to Figs. 4A to 4D, the two sequences are produced at 
different times but the two cameras that produce the respective two sequences are 
moving generally along the same path. The sequences may not have any spatial overlap 
or may have a partial overlap but still lack a degree of similarity required by the prior 
art. Such lack of similarity may result from differences in internal camera calibration, 
such as magnification, zoom and resolution, or from employing different sensing 
modalities. This lack of similarity may also result from changes in the scene, such as 
changes caused by the seasons, as illustrated in Figs. 4A to 4D, or due to differences in 
visibility between the two sequences. 

The mathematical treatment which follows employs an appreciation that 
despite there not necessarily being any common static visual information in the two 
sequences, the image to image dynamics in each of the two sequences are nevertheless 
related both spatially and temporally. This mathematical treatment utilizes the 
relationship between the dynamics to correlate the two sequences in time and space, 
without requiring understanding or interpretation of the features captured by the 
sequences. 

As will be described hereinbelow in detail, the image to image dynamics 
preferably are expressed by spatial transformations over time between images in each 
sequence. The problem of aligning sequences of images in time and space is thus 
reduced to a problem of determining the relationship between sequences of such 
transformations in time and in space. 

The- motions of the two cameras induce motions to the two sequences. The 
induced motions may be different for the two sequences but are correlated. Although the 
motions of the cameras are not known, the correlation between the induced motions 
serves to correlate between sequences both in time and in space. 

To correlate two images using the image to image dynamics, it is required 

that: 

( 1 ) Each of the images must be a part of a temporal sequence of images; 

(2) The two sequences must be produced during relative motion between 
cameras and a scene; 
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(3) The optical parameters of the two cameras must not change other than 
in a known manner throughout the production of the two sequences. 

The mathematical treatment which follows also imposes the following 
requirement in the examples presented therein; 
5 (4) The cameras producing the two sequences must have approximately 

the same center of projection throughout the production of the two sequences. 

The present invention does not require that any single image in one 
sequence of images have any spatial or visual similarity with any single image in the 
other sequence. Consequently and in addition, the present invention provides alignment 
1 0 between two images belonging to two sequences of images when the level of similarity 
between these images is insufficient to provide alignment by employing the prior art. 
The present invention normally does not require any prior internal or external camera 
calibration at any time. 

Each of the two prefeiTed embodiments of the present invention described 
1 5 hereinbelow is applicable to one of the following two situations: 

(a) A substantial portion of the scene is either planar or distant from the 
cameras producing the two sequences of images; 

(b) The scene is effectively three-dimensional. 

It is appreciated that the two situations each pertain to all the applications 
v 20 described hereinabove in accordance with Figs. 1 A to 4D. 

As shown and described hereinabove with reference to Figs. 6A to 6C, 
simple camera motion may not be sufficient to resolve the spatial and the temporal 
transformations between images of the two sequences. Therefore several different types 
of camera motions may be required. Three examples of such camera motion are shown 
25 in Figs, 5 A to 5C, it being understood that other, perhaps better, examples of camera 
motion may exist. An embodiment of the present invention described hereinbelow 
provides assessment of the required complexity of the motion. 

Reference is- now made to Fig. 7, which is a simplified illustration of a 
complex motion 700 of two cameras 702 and 704, fixed to each other, each camera 
30 producing a sequence of images of a portion of a scene 703, the sequences being 706, 
708, 710 and 712, 714 and 716 correspondingly, the portions of the scene photographed 
by the two cameras being non-overlapping. 
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Reference is now made to Fig. 8, which is a simplified illustration portions 
of two sequences 720 and 722 of images taken as shown in Fig. 7, wherein the two 
sequences are spatially related by a fixed and unknown inter-camera homography and 
temporally related by a fixed and unknown time shift At The symbols l u Tj, T'j and H 
are defined hereinbelow. 

Reference is now made to Fig. 9, which is a functional block diagram of a 
preferred sequence of steps of implementation of a preferred embodiment of the present 
invention. 

In step 900 two cameras 910 and 920 are employed to produce two 
sequences of images 930 and 940. 

Let S be a first sequence of images I ( produced by a first camera and let S' 
be a second sequence of images produced by a second camera, for example as shown 
and described in accordance with any of the Figs. 1A to 8 hereinabove, and wherein S 

=// /*HandSW/ /W 

The two* input sequences have an unknown temporal relation and an 
unknown spatial relation between them. 

In a preferred embodiment.of the present invention the spatial relation may- 
represent the fact that the two cameras are firmly attached to each other. 

In another preferred embodiment of the present invention the two cameras 
are mounted rigidly on the same device or rig, and moved jointly in space. 

In yet another preferred embodiment of the present invention the spatial 
relation may represent the feet that the cameras are moving in a generally identical 
trajectory but at different times. 

In an additional preferred embodiment of the present invention the spatial 
relation is produced by moving at least one cameras in a generally identical trajectory 
but in at least one of a different time and a different location, wherein there is a fixed 
relative position and a fixed relative orientation between the two cameras and wherein 
the optical .properties of each of the two cameras do not change between the images h 
and 7/+i and between /',• and 

In a further preferred embodiment of the present invention the spatial 
relationship is produced by moving at least one cameras in a generally identical 
trajectory but in at least one of a different time and a different location, wherein the 
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relative position and a relative orientation between the two cameras may change in a 
known way and wherein the optical properties of each of the two cameras may change 
between the images h and and between /'/ and / Vh in a known way. 

In a further preferred embodiment of the present invention the spatial 
5 relationship is produced by moving at least one camera at different times and/or in 
different locations, along generally different trajectories, wherein the at least two 
trajectories are correlated as if they were produced by two cameras that are mounted 
rigidly on the same device or rig, and moved jointly in space. 

The temporal relation between the two sequences represents the time when 
10 the two cameras were in the spatial relation as described hereinabove. 

In a preferred embodiment of the present invention the two sequences are 
produced by. employing synchronization equipment to synchronize between the two 
cameras. In this case the temporal relation between the two sequences represents the 
fact that the image 7} and the corresponding image I\ are produced together. 
15 In another preferred embodiment of the present invention the temporal 

relation between the two sequences represents the fact the image // and the 
corresponding image are produced with a fixed time delay between them. 

In yet another preferred embodiment of the present invention the temporal 
relation applies to image I t and a point in sequence S' that is between a corresponding /'/ 
20 and image I 

In yet another preferred embodiment of the present invention the temporal 
relation applies to two sequences produced employing two cameras employing two 
different frame rates. 

In a preferred embodiment of the present invention the temporal relation 
25 between the two sequences may represent the fact that the two images are the to be in 
the same "temporal relationship" if the cameras producing the sequences were in the 
same location In the same time. 

It is appreciated that temporal relation between sequences may exist even if 
the temporal relation between images may hold only for virtual images, that is, for 
30 images that have not been produced but could have been produced if the camera was 
producing an image at that time. 

It is also appreciated, that S and S' are not necessarily the original sequences, 
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but may by subsequences of the original sequences, where subsequence may represent, 
a portion of the field-of-view, temporal portion of the sequence, sub-sampling of pixels, 
and sub-sampling of frames. 

The sequences S and S' may be produced in several methods as described 
hereinbelow: 

The first and the second camera may have- a different 3D •orientation and 
move jointly in space, such as the camera pairs shown in Fig 7. In a preferred 
embodiment of the present invention none of the images contains any feature 
contained in any- of the images however, there is at least a partial temporal overlap 
between S and 5". 

Ln another preferred embodiment of the present invention the first camera 
and the second camera have at least partially overlapping fields of view, wherein images 
// comprises details that are not comprised in the overlapping parts of the corresponding 
images Vj and wherein the two cameras move jointly in space, such as the camera pair 
shown and described in accordance with Figs. 3A to 3D hereinabove. 

In yet another preferred embodiment of the present invention the sequence S 
and the sequence S' may be produced by the same camera moving in at least 
approximately the same pattern in space and at different times. 

Also in another preferred embodiment of the present invention the sequence 
S and the sequence S' may be produced by two cameras moving in at least 
approximately the same pattern in space and at different times. 

It is appreciated that the sequences S and iS" may be produced by any image 
producing instrument that produces at least two dimensional images of space, such as 
photographic camera, such as still film camera, digital camera, video camera or motion 
film camera, infra-red imaging equipment, radar, . x-ray, CAT-scan, mri and other 
electromagnetic imaging equipment, acoustic imaging equipment, such as ultrasound, 
sonar and sub-acoustic geophysical surveillance equipment and satellite remote sensing 
equipment. It is also appreciated that the two sequences S and S 1 may be produced by 
image producing instruments of the same type or of different types. 

It is appreciated that neither of the two sensors has to be calibrated. 

The temporal correspondence can be achieved electronically,, for example 
by employing Genlock, or recovered using the method describe hereinbelow with 
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reference to element 960 of Fig. 9. 

Furthermore, a method for computing temporal relation described 
hereinbelow can compute temporal relation at sub-frame time units. In this case a 
temporal relation is computed between a transformation T f and a transformation Tj+s 
wherein Tj+s is interpolated from 2} and Tj+\. The sequence of transformations Tj±s 
applies to virtual images Ij+s> wherein in the virtual images were not actually produced 
by the camera producing image sequence S\ but could have been produced by the 
camera if the images produced at the appropriate time points and are therefore 
interpolated representations of the scene. 

Reference is now made to step .990 of Fig. 9. It can now be assumed that the 
transformations and images are temporally synchronized. 

Let H be a spatial transformation function that transforms between a 
coordinate system of sequence S and a coordinate system of sequence S\ Applying the 
transformation H on the sequence S' results in the content of images /'/ in the 
coordinates of the sequence 5, as if the transformed sequence £" has been produced by 
the camera that has produced the sequence S. Therefore: 

p ' = H(p) where p 9 is the feature p of 7, in the coordinates of /',. 

In one preferred embodiment of the present invention, typically where the 
two cameras have approximately the same center of projection, if is a 3 x 3 inevitable 
matrix. 

Such H is denoted here as a "homography", the terms collination and 2D 
projective transformation may be equally used. In this case p and jc?' are given in 
homogeneous coordinates and the equation p' = H(p) becomes p f ~ Hp where 
= denotes equation up to a scale factor. 

In another preferred embodiment of the present invention, typically where 
the two cameras have a different center of projection and are firmly attached to the same 
rig H is a 4 x 4 inevitable Euclidean or inevitable projective matrix. 

However, the two sequences 5 and S do not share sufficient common 
feature or any other properties of the images. Therefore we cannot compute H from 
common properties of the images Instead H is computed by correlating between 
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properties of the temporal progressions within each sequence 

In a preferred embodiment of the present invention H is computed from the 
induced frame to frame transformation within each of the sequences S and S'. 

Let T be a first sequence of frame-to-frame transformations T t and let T' be 
a second sequence of frame-to-frame transformations T\ wherein T = Tu^.J^i and T* 
= r , |,...,r , //H .| and wherein 7/4.] s T/I;andr M & 7W 

In a preferred embodiment of the present invention, a large portion of the 
scene that is photographed by the two cameras, producing the sequences S and S\ is 
planar or distant from the cameras. In this case T t and T\ are 3x3 non-singular matrices. 

Let P be a three dimensional point in the planar or distant scene and let pi 
and/? be the homogeneous coordinates of the projection of the point P in the images h 
and respectively. It is noted that there is no need for P to appear in both or in any of 
the images // and V h that is, within the field of view of the cameras when images h or /'/ 
are produced Let p M and^pVi be the coordinates of the projection of the point P in 
images arid respectively. 

Then: . 

Pw s TfPi and pVn sT'p't 

and 

P'l^Hpi and p' m ^Hp^. 

and therefore Eq. (1) is: 

HTfPt m Hp M *p >l m T'p m T'lHpt 

and therefore Eq. (2) is: 
HT f & T'tH 

This operation is also referred to as "matrix composition". 
Since H is inevitable then Eq. (3) is: 
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T'i —SfHTfH' 1 where s is a frame dependent scale factor. 

Eq.(3) is true for all the images, that is, for any pair of temporally 
corresponding transformations T, and T h where i = This equation shows that 

there is a similarity relation between the two matrices 7) and T\ up to a scale factor. A 
matrix A is the to be "similar" to a matrix B if there exists an inevitable matrix M such 
that A = MBAf\ The term "conjugate matrices" can be used instead of "similar 
matrices". 

Let eig(A) = [Z 1,^2,^3]' be a 3x 1 vector containing, in decreasing order, 
eigenvalues of a 3 x 3 matrix A. Q l denotes the transpose vector. It is known in the art, as 
described for example in C. E. Pearson (ed.). Handbook of applied mathematics - 
Second Edition. Van Nostrand Reinhold Company, New York, 1983, pp 898, that: 
If A and B are similar matrices then they have the same eigenvalues: eig(A) = eig{B\ 
and, the eigenvalues of a scaled matrix are scaled: eig(sA) = s(eig(A)). 

Using these two facts and Eq. (3) we obtain Eq. (4) hereinbelow: 

eigCTd = Si eigCF/) where s< is the scale factor defined by Eq. (3). 

Equation (4) implies that eig{T^ and eig^T'd are "parallel". This gives rise 
to a measure of similarity between two matrices T, and T' h denoted by sim(T u T'd and 
presented in Eq. (5) hereinbelow: 

.^ >ra ~ i«w)iSfi))r '.*"""»"-'-• 

For each pair of the temporally corresponding transformations Ti and T\ in 
the sequences S and S' the eigenvalues eig(Ti) and eigiT'i) are first computed. The scale 
factor si which relates the eigenvalues eig{T^ and eigPT'i) is then estimated from Eq. (4). 
Eq. (4) is a set of three equations with one unknown and can be solved using least 
squares minimization. Alternatively, the input homographies Ti and T\ can be 
normalized to have determinant equal 1 and to avoid the need to compute 
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Once Si is estimated, Eq. (3) or Eq. (2) can be rewritten in the form of Eq. 
(6) hereinbelow: 

SiHTf-T'fH^O 

Equation (6) is linear in the unknown components of H. Rearranging the 
components of H in a 9 x 1 column vector h = lH n Hx2H X 2H 2 \H' n H 1 3H^HyzH^, Eq. 
(6) can be rewritten as a set of linear equations in h in the form of Eq. (7) hereinbelow: 



10 



Mjh = 0 



wherein M, is a 9 x 9 matrix defined by T h T\ and s f 
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and wherein 7 is the 3 x 3 identity matrix. 

Eq. (7) implies that each pair of corresponding transformations T t and T\ 
contributes 9 linear constrains in the unknown homography if (i.e., h ). 

The constraints from all the transformations T\,..,T n and T'u..,T' n can be 
combined into a single set of linear equations in h provided by Eq (8) hereinbelow: 



Ah= 0 



where A is a 9«x 9 matrix: A 



Equation (8) is a homogeneous set of line equations in h that can be solved 
in a variety of ways. In a preferred embodiment of the present invention h is solved by 
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computing the eigenvector which corresponds to the smallest eigenvalue of the matrix 
A'A. 

In another preferred embodiment of the present invention the scene for 
which the sequences S and S* have been produced is neither planar nor distant. In this 
5 case the temporal progression between any two consecutive images and If+\ is 
described by a fundamental matrix F h 

The fundamental matrix F, defines the relation between corresponding 
image points p t so that: 

10 if pt in // and pn-i in /*+! are corresponding image points, then 

p l i+\FiPi = 0 wherein Ft are each a 3 x 3 matrix of rank 2, 

Let F be a.sequence of fundamental matrices Ft wherein F = F\,..,F n 
1 5 Respectively, the temporal progression between any two consecutive images /'/ and I'm 
is described by F % h wherein F = F'^fn. 

Many methods for computing fundamental matrices are known in the prior 
art. One such method is taught by P.H.S. Torr and A. Zisserman in "Feature based 
methods for structure and motion estimation" in the proceedings of the Vision 
20 Algorithms Workshop pages 279-290, Corfu, 1999. Another method is taught by Z. 
Zhang, R. Deriche, O. Faugeras, and Q. Luong in "A robust technique for matching two 
uncalibrated images through the recovery of the unknown epipolar geometry'* in the 
journal of Artificial Intelligence, 78:87-1 19, 1995. 

The two sequences S and may not share sufficient common feature or any 
25 other properties of the images. Therefore the spatial and the temporal relations between 
the sequences S and S' are computed by correlating between properties of the temporal 
progressions F = F\ ,..JF n and F* - F\,..JF' n within each sequence . 

It is appreciated that neither of the two cameras has to be calibrated. 
It is appreciated that because the two cameras share the same center of 
30 projection the inter-camera transformation remains a homography H although the 
relations within each sequence are characterized by fundamental matrices, 

41 



WO 02/47031 



PCT/ILOl/01120 



In a preferred embodiment of the present invention, where the scene for 
which the sequences S and S> have been produced is neither planar nor distant, H is 
computed from properties of the temporal progression output as expressed by the 
sequences F and F' of the matrices F, and F' t . 

Each fundamental matrix F, can be decomposed into a homography + 
epipole as described by Eq. (9) hereinbelow:: 

Pi = [eijxT, wherein e, is the epipole relating frames 7, and I m ; 

wherein the matrix T, is the induced homography from /, 
to 7*1 via any plane, real or virtual; and 
wherein [-J, is the cross product matrix [v]* w = v x w . 

The homographies, T u ...,T n and T'\,...,T'„, and the epipoles e u ...,e n and 
• W* impose separate constraints on the inter-camera homography H. These 
15 constraints can be used separately or jointly to recover Has described, hereinbelow. 

(i) Homography-based constraints: 
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The homographies T u ...,T n and r,,...,7"„ extracted from the fundamental 

matrices F u ...,F n and F' ,F'„, respectively may correspond to different three 

dimensional planes. In order to apply the algorithm described above for the case of a 
planar or a distant scene, using these homographies, we have to impose 
plane-consistency across the two sequences, to guarantee that temporally corresponding 
homographies correspond to the same plane in the three dimensional world. 

In a preferred embodiment of the present invention the "Plane+Parallax" 
method which is well known in the prior art, is used to impose plane-consistency across 
and within the two sequences T U ...,T„ and T\,...,T',. The Plane+Paiallax method 
describes the image-to-image transformation using two components: homography and 
residual parallax.. The homography is associated with a selected plane and relates points 
on this plane between the two images. The parallax component describes the residual 
displacement of off-plane points. The "Plane+Parallax" method requires that a real 
physical planar surface be visible in all images I, and 
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One method for computing a Plane+Parallax transformation is taught by M. 
Irani, P., Anandan and D. Weinshall in "From reference frames to reference planes: 
Multi-view parallax geometry and applications" in the proceedings of the European 
Conference on Computer Vision, Freiburg, June 1998. 

Another method for computing a Plane+Parallax transformation is taught by 
R. Kumar, P. Anandan and K. Hanna in "Direct recovery of shape from multiple views: 
parallax based approach" in the proceedings. of the International Conference on Pattern 
Recognition, 1994. 

• Yet another method for computing a Plane+Parallax transformation is taught 
by Harpreet Sawhney in "3D geometry from planar parallax" in the proceedings of the 
IEEE Conference on Computer Vision and Pattern Recognition, June 1994. 

Another method for computing a Plane+Parallax transformation is taught by 
A. Shashua and N. Navab in "Relative afifine structure: Theory and application to 3D 
reconstruction from perspective views" in the proceedings of the IEEE Conference on 
Computer Vision and Pattern Recognition, pages 483-489, Seattle, WA, June 1 994. 

The homography part of the Plane+Parallax transformation is then used in 
the same manner as in the planar or distant case described hereinabove. 

In another preferred embodiment of the present invention the "threading" 
method, as described for example in S. Avidan and A. Shashua. Threading fundamental 
matrices. In European Conference on Computer Vision, 1998, which is also well known 
in the prior art, is used to impose plane-consistency across and within the two sequences 
7|,...,r ff and* r , i,...,r , w .. The "threading" method can impose plane-consistency within 
each sequence, even if no real physical plane is visible in any of the frames. 

It is appreciated that there are other methods in the prior art for computing a 
consistent set camera matrices. It is appreciated that the term camera matrix refers to a 4 
x 3 matrix, which is comprised of a 3 x 3 left minor, matrix and a 1 x 3 vector, wherein 
the 3 x 3 left minor matrix represents a 3 x 3 homography. The consistency of the 
camera matrices imposes plane-consistency on the 3x3 homographies. 

In a preferred embodiment of the present invention plane consistency 
between the two sequences is provided by applying the threading method on each of the 
two sequences S and iS", by initiating the threading method at frames which are known 
to simultaneously view the same real plane in both sequences. 
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It is appreciated that the two cameras can see different portions of the plane, 
allowing for non-overlapping fields of view, and need not see the plane at any of the 
other images. 

(ii) Epipole-based constraints: 

5 

In a preferred embodiment of the present invention the homography H is 
computed from epipoies e, and e ' h which are computed from the fundamental matrices 
■Fj and_F7 Given two images /, and I ^ an epipole denotes the spatial location of the 
center of projection of the camera for image in the coordinates system of image h 

10 The fundamental matrices F\....JF n and F'i,...^* provide a list of epipoies e\,...,e n and 
e'lvit-m wherein an epipole is the null space of a fundamental matrice Fi and an 
epipole e ) is the null space of a fundamental matrice FV 

It is appreciated that the epipoies e , and e-, can be computed by other 
methods than from the null space of the fundamental matrices F L and F' L respectively. 

15 There are many othei methods known in the prior art to find epipoies between two 
images. 

The epipoies e t and e*t are uniquely defined and therefore there is no issue 
of plane consistency. Since the two cameras have the same center of projection, then for 
any pair of images e } and e H satisfies the equation: 

20 

e e He h 

which yields Eqs. (10a and 10b) hereinbelow: 

25 («*,), and (Q 

Multiplying by the denominator and rearranging terms yields two new linear 
constrains on H for every pair of corresponding epipoies e { and e \ as described in Eq. 
(11) hereinbelow: 

30 
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h - 0 where 0' = [0, 0, 0]. 

2x9 

Therefore, every pair of temporally corresponding epipoles e t and e\ 
imposes two linear constraints on K The two sequences ei,...,e rt and e ' n provide a 
homogeneous linear system of 2n equations, wherein Eq(ll) is one pair of these 2n 
equations. 

The system of 2n equations can be solved for H in a variety of ways. 

In., a preferred embodiment of the present invention h is solved by 
computing the eigenvector which corresponds to the smallest eigenvalue. Theoretically, 
four pairs of corresponding epipoles e/ and e ) are sufficient, provided that no three 
epipoles are on the same line. 

Alternatively, # can be solved by adding the system of 2n equations to the 
set of linear equations in Eq. (8) which are imposed by the homographies. 

Reference is now made to step 960 of Fig. 9. 

In yet another preferred embodiment of the present invention the sequences 
S and 5" are not temporally synchronized. Namely, the image // in sequence S 
corresponds to image 77*. At in sequence S' and Ar is unknown. Therefore the 

transformation T f corresponds to transformation 7^ and not to T'u If time stamping, or 

similar additional information, is available for each image 7, and r t then synchronization 
can be recovered. When there is no additional information to recover the 
synchronization, the synchronization can be recovered in using the method hereinbelow. 

Given two unsynchronized sequences of transformations T u .„ 9 T n and 
7"|,...,7" OT , wherein 7) and T-^ are temporally corresponding transformations. 

According to Eq. (4) eig(Ti) || eig^T^ ) meaning that the 3 x 1 vectors of eigenvalues 

are parallel, namely there is an angle of 0° between the two vectors. Therefore the 
similarity measure simfTg^T^) of Eq. (5) is equal to 1, corresponding to cos(0). All 

pairs of temporally corresponding transformations T< t and T % utl must simultaneously 
satisfy this constraint. When the unknown temporal shift A t can be modeled by a set of 



0' e\ ( e >; 
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parameters A/, =f(U a/... an), then the parameters can be recovered by maximizing the 
following objective function presented hereinbelow as Eq. (12): 

5/M(a,...,a n )= ^sim^T^) 2 
i 

5 

For example, when the unknown At is a simple global temporal shift, it can be 
recovered by maximizing the following objective function presented hereinbelow as Eq. 
(12.a): 

10 SIM(At)= ^simPnT,^) 2 

I 

In a preferred embodiment of the present invention the maximization is 
performed by an exhaustive search over a finite range of valid time shifts A t. 

Alternatively, to address larger temporal shifts, a hierarchical search is 
applied. Coarser temporal levels are constructed by composing transformations to 
1 5 obtain fewer transformation between more distant frames. 

In still another preferred embodiment of the present invention the images in 
sequence S are produced at a different rate than the images of sequence S \ For example, 
when sequence S is produced by an NTSC camera at 30 frame per second and sequence 
5" is produced by a PAL camera at 25 frames per second. The ratio between the rate of 
20 recording of the two sequences S and S' is fixed and in the case of PAL and NTSC 
cameras it is 25:30 = 5:6. Therefore, the objective function that should be maximized 
for an PAL-NTSC pair of sequences as presented by Eq. (13) hereinbelow: 

SIM(At) -^M^r^T^T^ 
25 where 7*/ is the transformation from frame Ij to frame I Jm 

PAL and NTSC video cameras produce each image as pair of fields, 
wherein the first field comprises the odd lines of the image and the second field 
comprises the even lines of the picture. The fields are produced in a rate that is double 
30 the rate of the images, or frames, that is at 60 and 50 fields per second respectively. The 
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method described above with respect to Eq. (12.a) can recover synchronization to the 
field level. Sub-field accuracy can be recovered by interpolating the values of SIM(Af) 
obtained at discrete time shifts. 

In a preferred implementation of the present invention the transformations 
are interpolated according to the computed temporal correspondence and SIM(At) is 
re-estimated, in an iterative manner, to increase the temporal accuracy. 

In order to solve H it is required to have a sufficient number N min of 
corresponding pairs of frame-to-frame transformations T ( and T\ 

In a preferred embodiment of the present invention the number N min is 
assessed by examining via the similarity equation Eq. (3) the number of constraints 
imposed on H by a single pair of transformations Ti and T' h 

The scale factor s f can be extracted directly fronrr, and. T't as described 
hereinabove with reference to element 950 of Fig. 9. and therefore can be omitted from 
Eq. (3). Therefore, the number of constraints imposed on H by a single pair of 
transformations via the similarity equation Eq. (3) can be estimated by examining the 
number of constraints imposed on H by an equation of the form G = HBH X where B = 
Tj and G r= T\ A general analysis of matrix equations of the form GH = HB may be 
found in The theory of matrices. Chapter VIII, by F. R. Gantmakher from Chelsea Pub., 
New York, 1959. 

The following notations are used hereinbelow: 

denote by X\, X 2i X 3 the eigenvalues of the matrix B in decreasing order 
* M * IM); 

denote by u hl% u hl , u b3 the corresponding eigenvectors with unit norm 

(ll^ilI = !l^ 2 ll = ll" A3 ll); 

denote by r y the algebraic multiplicity of the eigenvalue kj; and 

denote by Vj= { v e R n : Bv -\jv } the corresponding eigen subspace. 

It is noted that: 

■If Xi ? X 2 ? X 3 then the algebraic multiplicity of all eigenvalues is 1 (r y =1); 
.If X\ = A3 ? ^3 then the algebraic multiplicity of X\ and X 2 is 2, and the 
algebraic multiplicity of X 3 is 1 (n = r 2 = 2 and r 3 = 1); and that 

If X\ = X.2 =X 3 then the algebraic multiplicity of X\ , X 2 , and X 3 is 3 
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(n = r 2 = r 3 =3). 

Similar matrices, also known as conjugate matrices, such as B and G, have 
the same eigenvalues but different eigenvectors. The eigenvectors of the similar 
matrices are related by H. If u b is an eigenvector of B with corresponding eigenvalue X, 
then H\x h is an eigenvector of G with the same eigenvalue X so that G(ffu b ) = A(Hu b ). 
The same applies for eigen subspaces, wherein if V is an eigen " subspace of B 
corresponding to an eigenvalue X, then H(V) is an eigen subspace G with the same 
eigenvalue X The number of constraints imposed on H by B and G is therefore 
investigated according to the dimensionality of the eigen subspaces of B and G. Let Fbe 
the eigen subspace corresponding to an eigenvector u b of B. Three possible cases are 
valid, one case for each.possible dimensionality of V, i.e., dim(V) = 1, 2, 3. 

Case I: dim(V)=\. 

This case typically occurs when all three eigenvalues are distinct. The case 
can also occur when some eigenvalues have algebraic multiplicity of two or three. In all 
these cases, Fis spanned by the single eigenvector Ub. Similarly H(V) is spanned by the 
eigenvector u g of G. Therefore Eq. (13): 

Hu h = au g wherein a is an unknown scale factor 



Eq. (13) provides three linear equations in H and one new unknown a and 
therefore Eq. (13) provides two new linearly independent constrains on H. 
Case II: dim(V) = 2. 

This case typically occurs in one of the following two cases: 

(a) when there exists an eigenvalue with algebraic multiplicity two; or 

(b) when there is only one eigenvalue with algebraic multiplicity three. 
However, the eigen subspace spanned by all eigenvectors has 

dimensionality of two. For example, a homography defined by pure shift (Ax, Ay) has 
the form: 



H 



1 0 Ax 
0 1 Ay 
0 0 1 
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10 



15 



This matrix has a single eigenvalue X, = X 2 = X 3 - 1 with algebraic 
multiplicity three. The corresponding eigen subspaces has dimensionality 2. It is 
spanned by two linearly independent eigenvetors [1,0,0]' and [0,1,0]'. 

When dim(V) - 2 the two eigenvectors span V, such that, for example, 
w.l.o.g., u bl and u b2 . Therefore every linear combination of u„, and u b2 is also an 
eigenvector of B with the same eigenvalues. Similarly, every linear combination of u g , 
and u g2 is an eigenvector of G with the same eigenvalue. Therefore Eq. (14): 

Hu hj = ayiigi + fljugj where ib a y and Bj are unknown scalars (/ = 1 ,2) 



Therefore, each of the two eigenvectors u b , and u b2 provides 3 linear 
equations and 2 new unknowns. Therefore, together the two eigenvectors u bl and u bI 
provide 2 new linear constraints on H. 
Case III: dim{V) = 3. 

In this case any vector is an eigenvector and all the eigenvector have the 
same eigenvalues A. This is the case when B « G . XI are the identity transformation 
up to a scale factor, which means that there is no motion. In this case B and G provide 
no additional constraints on H. 

The number of constraints imposed on H were counted hereinabove for a 
20 single eigen subspace. In order to count the total number of linear constraints that B and 
G impose on H every possible combination of eigen subspaces is analyzed according to 
the algebraic multiplicity of the eigenvalues. There are three such combinations: 

1 . X, * Xj * X k , which implies that: 

V, * Vj , * V k , and 
25 dim(Vt) = dim(Vj) = dim(V k ) = 1 . 

2. X/ ~ Xj X/f , which implies that V t = Vj # V k . 
There are two such cases: 

(a) dim(Vi = V/) - 2 and dim(Vk) = 1 . 

(b) dim(Vi = P£) = 1 and rfzm(J^) = 1. 
30 3.X, = X y = X* 

In this case there is only a single eigen subspace V= V t = Vj= V k , which 
dimensionality may 1, 2, or 3. 
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The following table summarizes the number of linearly independent 
constraints of each of the above cases: 





Eigenvalues 


Eigen Subspaces 


# of Linearly 


Case 


Algebraic 


dimensionality 


independent 




Multiplicity 




constraints 


0) 


X/ 5* Xj * %k 


1^1=1^1=1.^1=1 


6 


(2.a) 


Xf*=Xj s* Xk 


|F,-*5|-2.|Fy-l 


4 


(2.b) 


Xf—Xj Xk 


w-^-i. im=i 


4 


(3.a) 


X j — Xj— Xk 




2 


(3.b) 


X/ = Xj — Xk 


1^=^=^1 = 2 


2 


(3.c) 


X , ~ Xy = Xk 


\ Vi = V j=V^ = 3 ■ . 


0 



5 Therefore, when 5 and G have either two or three distinct eigenvalues, 

which is typical of general frame-to-frame transformations, then two independent pairs 
of transformations suffices to uniquely determine H, since each pair of transformations 
imposes 4 to 6 linearly independent constrains, and, according to the prior art, 8 
independent linear constrains suffice to uniquely resolve if, up to an arbitrary scale 
10 factor. In a preferred embodiment of the present invention all available constrains from 
all pairs of transformation, for increased numerical stability. 

In a preferred embodiment of the present invention two cameras employed 
to produce two input sequences, preferably several seconds long to provide significant 
motion. 

15 Referring to step 950, a frame-to-frame input transformations, T\ 9 ...,T n and 

r|„..,rv are preferably extracted, for the case of the planar or distant scene, using the 
method described in, for example, M. Irani, B. Rousso and S. Peleg. Computing 
occluding and transparent motions. International Journal of Computer Vision, 12(1):5- 
16, January 1994. 

20. Reference is now made to step 970 of Fig. 9. Step 970 comprises methods 

of robust statistics to enhance the accuracy of the transformations 7/ and TV In a 
preferred embodiment of the present invention step 970 comprises two optional 
methods: 
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(a) Outlier rejection, designated by reference numeral 972; and 

(b) Coordinate renormalization, designated by reference numeral 974. 

In step 972 inaccurate frame-to-frame transformations T { are preferably 
pruned out by employing any of the two outlier detection mechanisms known in the 
5 prior art as described hereinbelow: 

(i) In a preferred embodiment of the present invention, the transformation 
between successive frames within each sequence are computed in both directions. The 
distance of the composed matrix T i T i Reverx from the identity matrix in the image space is 
computed, in terms of the maximal residual misalignment of pixels. 

(ii) .Alternatively, the similarity criterion of Eq. (5) is employed to verify 
the degree of "similarity " between T f and 7 V An unreliable pair of transformations can 

1 5 thus be pruned out. 

In step 974, optionally, any one of two methods of coordinate 
renormalization known in the prior art is used as described hereinbelow: 

(a) The input matrices are re-normalized to increase the numerical 
stability, as described for example in Richard I, Hartley. In Defence of the 8-point 

20 Algorithm. In Pattern Recognition and Machine Intelligence 19 (6) June pages 580-593 
1997. 

(b) Input matrices are normalized so that the rows of M has 
approximately the same norm, using the heuristic provided in for example, Gene Golub 
and Charles Van Loan. Matrix Computations. The Johns Hopkins University Press, 

25 Baltimore and London, pp. 123-127, 1989. 

Reference is now made to step 980 of Fig. 9. 

In a preferred embodiment of the present invention, when the 
frame-to-frame transformations are too. small, temporal sub-sampling of the sequences 
can provide more significant transformations between successive frames. Preferably, 
30 temporal sub-sampling is done after recovering the temporal synchronization, in order 
to guaranty that temporally corresponding frames are sampled from the two video 
sequences. 
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In another preferred embodiment of the present invention the three 
dimensional joint motions of the two cameras are planned in advance to increase the 
accuracy of the computed result. 

For example, in a preferred embodiment of the present invention input data 
are sequences of transformation T, and T' L . The transformation T, and T' t are extracted 
from two video sequences using one of a variety of algorithms provided in the prior art. 
The reliability of the input data also depends on the specific algorithm used to extract 
the input transformations. Employing a rough approximation to the accuracy, of the 
different transformations T, and T', it is possible to analyze the combinations of the 
transformations T, and T' t that participate in the matrix M (Eq.7). Hence by analyzing 
the pure 6 motions, comprising thee translation and three rotations, the stability of the 
derived equations can be predicted. It is appreciated that a higher stability of the derived 
equations results in a higher accuracy of the alignment between the sequences. For . 
example, for the method described, for example, in M. Irani, B. Rousso and S. Peleg. 
Computing occluding and transparent motions. International Journal of Computer 
Vision, 12(1):5-16, January 1994, the accuracy of the projective parameters r 3 ., and 
73,2 is poor, the accuracy of the translation parameters T u and T v is good, and the. 
other parameters have fair accuracy. Therefore it is predicted that rotation within the 
image plane produces the most reliable set of equations. 

It is appreciated that various features of the invention which are, for clarity, 
described in the contexts of separate embodiments may also be provided in combination' 
in a single embodiment. Conversely, various features of the invention which are, for 
brevity, described in the context of a single embodiment may also be provided 
separately or in any suitable subcombination. 

It will be appreciated by persons skilled in the art that the present invention 
is not limited to what has been particularly shown and described hereinabove. Rather, 
the scope of the present invention is defined only by the claims that follow: 
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CLAIMS 

1 . A method for computing at least one of spatial and temporal relationships 
between at least first and second sequences of representations having respective first 

5 and second temporal progressions, the method comprising: 

employing said first and second temporal progressions to obtain at least one 
of said spatial and temporal relationships. 

2. A method for computing at least one of spatial and temporal relationships 
10 between at least first and second sequences of representations having respective first 

and second temporal progressions, the method comprising: 

employing only said first and second temporal progressions to obtain at least . 
one of said spatial "and temporal relationships. 

15 3. A method according to claim 1 or claim 2 and also comprising: 

computing said first and second temporal progressions which are then 

employed to obtain at least one of said spatial and temporal relationships. 

.* 

4. A method according to any of the preceding claims and wherein said 
20 • representations comprise visual representations. 

5. "" A method according to any of the preceding claims and wherein said 
representations comprise images. 

25 6, A method according to any of the preceding claims and wherein said 

representations-comprise at least three-dimensional objects. 

7. A method according to any of the preceding claims and wherein said 
employing obtains said spatial relationship. 

30 

8. A method according to any of the preceding claims and wherein said 
employing obtains said temporal relationship. 
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9- A method according to any of the preceding claims and wherein said 
employing obtains said spatial relationship and said temporal relationship 

10- A method according to any of the preceding claims and wherein said 
employing obtains said at least one of spatial and temporal relationships in the absence 
of a spatial overlap between said representations 

A method according to any of the preceding claims and wherein said 
employing obtains said at least one of spatial and temporal relationships in the absence 
of common spatial information between said-representations. 

12 A method according to any of the preceding claims and wherein said 

employing obtains said at least one of spatial and temporal relationships in the absence 
of common information between individual ones of said representations belonging to 
different ones of said at least first and second sequences. 

13. .A method according to any of the preceding claims and wherein said 
employing obtains said at least one of spatial and temporal relationships in the absence 
of common information between any individual ones of said representations belonging 
to different ones of said at least first and second sequences. 

14. A method according to any of the preceding claims and wherein said spatial 
relationship comprises at least one parameter of a geometric transformation between 
said at least first and second sequences of representations. 

15. A method according to claim 14 and wherein said spatial relationship 
includes a 2-dimensional projective transformation. 

16. A method according to claim 14 and wherein said spatial relationship 
includes a 3 -dimensional projective transformation. 
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17. A method according to claim 14 and wherein said spatial relationship 
includes a fundamental matrix. 

18. A method according to claim 14 and wherein said spatial relationship 
5 includes a 2-dimensional parametric transformation. 

19. A method according to claim 14 and wherein said spatial relationship 
includes an at least 3 -dimensional parametric transformation. 

10 20. A method according to claim 14 and wherein said spatial relationship 

includes a 2-dimensional non-parametric transformation. 

21. A method according to claim 14 and wherein said spatial relationship 
includes an at least 3-dimensional non-parametric transformation. 

15 

22. A method according to any of claims 14-21 and wherein said spatial 
relationship comprises a spatial relationship between individual ones of said 
representations belonging to different ones of said at least first and second sequences. 

20 23. A method according to any of claims 14-21 and wherein said spatial 

relationship comprises a spatial relationship between individual ones of said 
representations belonging to different ones of said at least first and second sequences, at 
least one of said individual ones of said representations being an interpolated 
" representation. . 

25 

24. A method according to any of claims 14-21 and wherein said spatial 

relationship comprises a spatial relationship between individual ones of said 
representations belonging to different ones of said at least first and second sequences, 
one of said individual ones of said representations being an interpolated representation 
30 and the other . of said individual ones of said representations being a non-interpolated 
representation. 
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25. A method according to any of the preceding claims and wherein said 
temporal relationship comprises at least one parameter of a temporal transformation 
between said at least first and second sequences of representations. 

26. A method according to claim 25 and wherein said temporal relationship 
includes a time shift between said at least first and second sequences of representations. 

27. A method according to claim 25 and wherein said temporal relationship 
includes an affine transformation in time. 

28. A method according to claim 25 and wherein said temporal relationship 
includes a parametric transformation in time. 

29. A method according to claim 25 and wherein said temporal relationship 
includes a non-parametric transformation in time. 

30. A method according to any of claims 25 - 29 and wherein said temporal 
relationship comprises a temporal relationship between individual ones of said 
representations belonging to different ones of said at least first and second sequences. 

31. A method according to any of claims 25 - 29 and wherein said temporal 
relationship comprises a temporal relationship between individual ones of said 
representations belonging to different ones of said at least first and second sequences, at 
least one of said individual ones of said representations being an interpolated 
representation. 

32. A method according to any of claims 25 - 29 and wherein said temporal 
relationship comprises a temporal relationship between individual ones of said 
representations belonging to different ones of said at least first and second sequences, 
one of said individual ones of said representations being an interpolated representation 
and the other of- said individual ones of said representations being a non-interpolated 
representation. 
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33. A method according to any of claims 23, 24, 31 and 32 and wherein said 
interpolated representation is interpolated in time. 

34. A method according to any of claims 23, 24, 3 1, 32 and 33 and wherein said 
interpolated representation is interpolated in space. 

35. A method according to any of the preceding claims and wherein said first 
and second temporal progressions comprise ordered intra-sequence 
representation-to-representation transformations. 

36. A method according to any of the preceding claims and wherein said first 
and second temporal progressions comprise ordered intra-sequence 
representation-to-representation transformations resulting from relative motion between 
sensors and a scene. 

37. A method according to either of claims 35 and 36 and wherein said 
intra-sequence representation-to-representation transformations include 2-dimensional 
projective transformations. 

38. A method according to either of claims 35 and 36 and wherein said 
intra-sequence representation-to-representation transformations include 3-dimensional 
projective transformations. 

39. A method according to either of claims 35 and 36 arid wherein said 
intra-sequence representation-to-representation transformations include fundamental 
matrices. 

40. A method according to either of claims 35 and 36 and wherein said 
intra-sequence representation-to-representation transformations include 2-dimensional 
parametric transformations 
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41 A method according to either of claims 35 and 36 and wherein said 

intra-sequence representation-to-representation transformations include at least 
3 -dimensional parametric transformations 

5 42 A method according to either of claims 35 and 36 and wherein said 

intra-sequence representation-to-representation transformations include 2-dimensional 
non-parametric transformations 

43 A method according to either of claims 35 and 36 and wherein said 
10 intra : sequence representation-to-representation transformations include at least 

3 -dimensional non-parametric transformations 

44 A method according to either of claims 35 and 36 and wherein said 
intra-sequence representation-to-representation transformations include camera 

1 5 matrices. 

45. A method according to any of the preceding claims and wherein said 
employing comprises: 

correlating said first and second temporal progressions. 

20 

46. A method according to any of the preceding claims and wherein said 
employing comprises: 

equating properties of said first temporal progression and said second 
temporal progression. 

25 

47. A method according to any of the preceding claims and wherein said 
employing comprises: 

correlating properties of said fiTSt temporal progression and said second 
temporal progression. 

30 

48. A method according to any of the .preceding claims 35 - 47 and wherein said- 
employing comprises: 
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equating a sequential application of at least one of said intra-sequence 
representation-to-representation transformations of said first temporal progression and 
an unknown said spatial relationship between said at least first and second sequences 
with a sequential application of said unknown spatial relationship between said at least 
first and second sequences and at least one of said intra-sequence 
representation-to-representation transformations of said second temporal progression. 

49. A method according to any of the preceding claims 35 - 47 and wherein said 
employing comprises: 

equating a composition of at least one of said intra-sequence 
representation-to-representation transformations of said first temporal progression and 
an unknown said spatial relationship between said at least first and second sequences 
with a composition of said unknown spatial relationship between said at least first and 
second sequences and at least one of said intra-sequence 

representation-to-representation transformations of said second temporal progression. 

50. A method according to any of the preceding claims 35-47 and wherein said 
employing comprises: 1 

obtaining an unknown said spatial relationship between said at least first and 
second sequences by equating a sequential application of at least one of said 
intra-sequence representation-to-representation transformations of said first temporal 
progression and said unknown spatial relationship between said at least first and second 
sequences with a sequential application of said unknown spatial relationship between 
said at least first and second sequences and at least one of said intra-sequence 
representation-to-representation transformations of said second temporal progression. 

51. A method according to any of claims 46 - 50 and wherein equating includes 
equating up to a scale factor. 

52. A method according to any of claims 35 - 51 and wherein said 
intra-sequence representation-to-representation transformations include multiple simple 
motions taking place at least partially at different times. 
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53. A method according to any of claims 35 - 52 and wherein said 
intra-sequence representation-to-representation transformations include multiple 
combinations of multiple simple motions taking place at least partially at different 
times. 

54. A method according to any of claims 35 - 53 and wherein said 
intra-sequence representation-to-representation transformations include multiple 
complex motions taking place at least partially at different times. 

55. " - ' A method according to any of claims 35 - 54 and wherein said 
intra-sequence representation-to-representation transformations include multiple 
combinations of multiple complex motions taking place at least partially at different 
times. 

56. A method according to any of the preceding claims and wherein said first 
and second temporal progressions comprise ordered intra-sequence 
representation-to-representation transformations at least some of which result from 
relative motion between sensors and a scene. 

57 - A method according to any of the preceding claims and wherein said 

employing - uses multiple combinations of intra-sequence 
representation-to-representation transformations. 

58. A method according to any of the preceding claims and wherein said spatial 
relationship between said at least first and second sequences of representations results 
from an acquisition relationship, between first and second sensors acquiring respective 
said at least first and second sequences, being fixed over time. 

59. A method according to any of the preceding claims and wherein* said spatial 
relationship between said at least first and second sequences of representations results 
from an acquisition relationship, between first and second sensors acquiring respective 
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said at least first and second sequences, changes in a known way over time. 

60. A method according to claim 58 or claim 59 and wherein said acquisition 
relationship comprises: 

5 relative position; 

relative orientation; and 

relative internal sensor parameters. 

61 . A method according to claim 58 and wherein said acquisition relationship is 
10 known. 

62. A method according to claim 58 and wherein said acquisition relationship is 
not known. 

15 63 . A method according to any of the preceding claims and wherein said at least 

first and second sequences are acquired generally at the same time. 

64. A method according to any of the preceding claims and wherein said at least 
first and second sequences are acquired generally at different times. 

20 

65. A method according to any of the preceding claims and wherein said at least 
first and second sequences represent measurements from the same scene. 

66. A method according to any of the preceding claims and wherein said at least 
25 first and second sequences represent measurements from different portions of the same 

scene. 

67. A method according to any of the preceding claims and wherein said at least 
first and second sequences represent measurements from different overlapping portions 

30 of the same scene. 



68. 



A method according to any of the preceding claims and wherein said at least 
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first and second sequences represent measurements from different non-overlapping 
portions of the same scene. 

69. A method according to any of the preceding claims and wherein said at least 
5 first and second sequences represent measurements from different scenes. 

70. - A method according to any of the preceding claims 65 - 69 and wherein said 
scene is two-dimensional. 

10 71. A method according to any of the preceding claims 65 -69 and wherein said 

scene is at least three-dimensional. 

72. . A method according to any of the preceding claims 65 - 69 and wherein said 
scene is static. 

15 

73 . A method according to any of the preceding claims 65 - 69 and wherein said 
scene is dynamic. 

74. A method according to any of claims 65 - 73 and wherein said 
measurements are generally the same for each sensor. 

75. A method according to any of claims 65 - 73 and wherein said 
measurements are generally different for each sensor. 

76. A method according to any of claims 65 - 75 and wherein said 
measurements include at least one of : 

illumination, heat, radiance, electromagnetic radiation, color, distance, 
density, sound and speed. 

77. A method according to any of the preceding claims and also comprising 
employing at least one of said spatial and temporal relationships for sequence fusion. 
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78. A method according to any of the preceding claims and also comprising 

employing at least one of said spatial and temporal relationships for alignment of 
sequences obtained at different zooms. 



79. A method according to any of the preceding claims and also comprising 
employing at least one of said spatial and temporal relationships for surveillance. 

80. A method according to any of the preceding claims and also comprising 
employing at least one of said spatial and temporal relationships for generation of 
wide-screen movies from multiple at least partially non-overlapping narrow field of 
view movies. 



81. A method according to any of the preceding claims and also comprising 
employing at least one of said spatial and temporal relationships for image fusion. 

82. A method according to any of the preceding claims and also comprising 
employing at least one of said spatial and temporal relationships for integrating 
information contained in said at least first and second sequences. 

83. A method according to any of the preceding claims and also comprising 
employing at least one of said spatial and temporal relationships for alignment of 
images obtained at different zooms. 



84. A method according to any of the preceding claims and also comprising 
employing at least' one of said spatial and temporal relationships for comparing 
information contained in said at least first and second sequences. 

85. A method according to any of the preceding claims and also comprising 
employing at least one of said spatial and temporal relationships for finding differences 
between information contained in said at least first and second sequences. 



86. A method according to any of the preceding claims and also comprising 
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employing at least one of said spatial and temporal relationships for finding differences 
between information contained in said at least first and second sequences relating to the 
same scene at different times. 



87. A method according to any of the preceding claims and also comprising 
employing at least one of said spatial and temporal relationships for integrating 
information contained' in said at least first and second sequences and thereby providing 
an information output which exceeds limitations of individual sensors. 

88. A method according to any of the preceding claims and wherein said 
employing comprises: 

comparing properties of said first temporal progression and said second 
temporal progression. 

89. A system for computing at least one of spatial and temporal relationships 
between at least first and second sequences of representations having respective first 
and second temporal progressions, the system comprising: 

input functionality for receiving at least one of said first sequence of 
representations and said first temporal progression and at least one of said second 
sequence of representations and said second temporal progression; and 

computation functionality employing at least one of said first sequence of 
representations and said first temporal progression and at least one of said, second 
sequence of representations and said second temporal progression to obtain at least one 
of said spatial and temporal relationships. 

90. A system for computing at least one of spatial and temporal relationships 
between at least first and second sequences of representations having respective first 
and second temporal progressions, the system comprising: 

input functionality for receiving said first and second temporal progressions; 

and 

computational functionality employing said first ' and second temporal 
progressions to obtain at least one of said spatial and temporal relationships. 
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91 . A system according to claim 89 and wherein said computation functionality 
includes functionality for computing said first and second temporal progressions which 
are then employed to obtain at least one of said spatial and temporal relationships. 

92. A system according to claim 90 and also comprising temporal progression 
computation functionality for computing said first and second temporal progressions 
and supplying them to said input functionality. 

93. A~ system according to any of claims 89 - 92 and wherein said 
representations comprise visual representations, 

94. A system according to any of claims 89 r 93 and wherein said 
representations comprise images. 

95. A system according to any of claims 89 - 94 and wherein said 
representations comprise at least three-dimensional objects. 

96. A system according to any of claims 89 - 95 and wherein said computation 
functionality includes functionality for obtaining said spatial relationship. 

97. A system according to any of claims 89 - 96 and wherein said computation 
functionality includes functionality for obtaining said temporal relationship. 

98. A system according to any of claims 89 - 97 and wherein said computation 
functionality includes functionality for obtaining said spatial relationship and said 
temporal relationship 

99. A system according to any of claims 89 - 98 and wherein said computation 
functionality includes functionality for obtaining said at least one of spatial and 
temporal relationships in the absence of a spatial overlap between said representations 
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100. A system according to any of claims 89 - 99 and wherein said computation 

functionality includes functionality for obtaining said at least one of spatial and 
temporal relationships in the absence of common spatial information between said 
representations. 



101. A system according to any of claims 89 - 1 00 and wherein said computation 
functionality includes functionality for obtaining said at least one of spatial and 
temporal relationships in the absence of common information between individual ones 
of said representations belonging to different ones of said at least first and second 
sequences. 

102. A system according to any of claims 89 - 101 and wherein said computation 
functionality includes functionality for obtaining said at least one of spatial and 
temporal relationships in the absence of common information between any individual 
ones of said representations belonging to different ones of said at least first and second 
sequences. 

103. A system according to any of claims 89 - 102 and wherein said spatial 
relationship comprises at least one parameter of a geometric transformation between 
said at least first and second sequences of representations. 

104. A system according to claim 103 and wherein said spatial relationship 
includes a 2-dimensional projective transformation. 

105. A system according to claim 103 and wherein said spatial relationship 
includes a 3-dimensional projective transformation. 

106. A system according to claim 103 and wherein said spatial relationship 
includes a fundamental matrix. 

107. A system according to claim 103 and wherein said spatial relationship 
includes a 2-dimensional parametric transformation. 
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108. A system according to claim 103 and wherein said spatial relationship 
includes an at least 3 -dimensional parametric transformation. 

109. A system according to claim 103 and wherein said spatial relationship 
includes a 2-dimensional non-parametric transformation. 

110. A system according to claim 103 and wherein said spatial relationship 
includes arrat least 3-dimensionaI non-parametric transformation. 

111. . A. system according to any of claims 103 - 110 and wherein said spatial 
relationship comprises a spatial relationship between individual ones of said 
representations belonging to different ones of said at least first and second sequences. 

112. A system according to any of claims 103 - 110 and wherein said spatial 
relationship comprises a spatial relationship between individual ones of said 
representations belonging to different ones of said at least first and second sequences, at . 
least one of said individual ones of said representations being an interpolated 
representation. . 

113. A system- according to any of claims 103 - 110 and wherein said spatial 
relationship' comprises a spatial relationship between individual ones of said 
representations belonging to different ones of said at least first and second sequences, 
one of said individual ones of said representations being an interpolated representation 
and the other of said individual ones of said representations being a non-interpolated 
representation. 

114. A system according to any of claims 89-113 and wherein said temporal 
relationship comprises at least one parameter of a temporal transformation between said 
at least first and second sequences of representations. 

115. A system according to claim 114 and wherein said temporal relationship 
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includes a time shift between said at least first and second sequences of representations. 

116. A system according to claim 114 and wherein said temporal relationship 
includes an affine transformation in time. 

117. A system according to claim 114 and wherein said temporal relationship 
includes a parametric transformation in time. 

1 18. A system according to claim 1 14 and wherein said temporal relationship 
includes a non-parametric transformation in time. 

119. A system according to any of claims 1 14 - 1 18 and- wherein said temporal 
relationship comprises a temporal relationship between individual ones of said 
representations belonging to different ones of said at least first and second sequences. 

120. A system according to any of claims 1 14 - 1 18 and wherein said temporal 
relationship comprises a temporal relationship between individual ones of said 
representations belonging to different ones of said at least first and second sequences, at 
least one of said individual ones of said representations being an interpolated 
representation. 

121. A system according to any of claims 1 14 - 1 18 and wherein said temporal 
relationship comprises a temporal relationship between individual ones of said 
representations belonging to different ones of said at least first and second sequences, 
one of said individual ones of said representations being an interpolated representation 
and the other of said individual ones of said representations being a non-interpolated 
representation. 

122. A system according to any of claims 112, 113, 120 and 121 and wherein 
said interpolated representation is interpolated in time. 



123. 



A system according to any of claims 112, 113, 120, 121- and 122 and 
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wherein said interpolated representation is interpolated in space. 

124. A system according to any of claims 89 - 123 and wherein said first and 
second temporal progressions comprise ordered intra-sequence 

5 representation-to-representation transformations. 

125. A system according to any of claims 89 - 124 and wherein said first and 
second temporal progressions comprise ordered intra-sequence 
representation-to-representation transformations resulting from relative motion between 

10 sensors and a scene. 

1*26. A system according to either of claims 124 and 125 and wherein said 

intra-sequence representation-to-representation transformations include 2-dimensional 
projective transformations. 

15 

127. A system according to either of claims 124 and 125 and wherein said 

intra-sequence representation-to-representation transformations include 3 -dimensional 
projective transformations. 

20 128. A system according to either of claims 124 and 125 and wherein said 

intra-sequence representation-to-representation transformations include fundamental 
matrices. 

129. A system according to either of claims 124 and 125 and wherein said 
25. intra-sequence representation-to-representation transformations include 2-dimensional 

parametric transformations 

130. A system according to either of claims 124 and 125 and wherein said 
intra-sequence representation-to-representation transformations include at least 

30 3-dimensional parametric transformations 

131. A system according to either of claims 124 and 125 and wherein said 

69 



WO 02/47031 



PCT/IL01/01120 



intra-sequence representation-to-representation transformations include 2-dimensional 
non-parametric transformations 

132. A system according to either of claims 124 and 125 and wherein said 
intra-sequence representation-to-representation transformations include at least 
3-dimensibnaI non-parametric transformations 

133. A system according to either of claims 124 and 125 and wherein said 
intra-sequence representation-to-representation -transformations -include camera 
matrices. 

134. A system according to any of claims 89 - 1 33 and wherein said computation 
functionality comprises: 

correlating said first and second temporal progressions. 

135. A system according to any of claims 89 - 1 34 and wherein said computation 
functionality comprises: 

equating properties of said first temporal progression and said second 
temporal progression. 

136. ..A system according to any of claims .89 - 1 35 and wherein said computation 
functionality comprises: 

correlating properties of said first temporal progression and said second 
temporal progression. 

137. A system according to any of claims 89 - 136 and wherein said computation 
functionality comprises: 

equating a sequential application of at least one of said intra-sequence 
representation-to-representation transformations of said first temporal progression and 
an unknown said spatial relationship between said at least first and second sequences 
with a sequential application of said unknown spatial relationship between said at least 
first and second sequences and at least one of said intra-sequence 
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representation-to-representation transformations of said second temporal progression. 

138. A system according to any of claims 89 - 137 and wherein said computation 
functionality comprises: 

equating a composition of at least one of said intra-sequence 
representation-to-representation transformations of said first temporal progression and 
an unknown said spatial relationship between said at least first and second sequences 
with a composition of said unknown spatial relationship between said at least first and 
second sequences and at least one of said- intra-sequence 
representation-to-representation transformations of said second temporal progression. 

139. A system according to any of claims 89 - 138 and wherein said computation 
functionality comprises: 

obtaining an unknown said spatial relationship between said at least first and 
second sequences by . equating a sequential application of at least one of said 
intra-sequence representation-to-representation transformations of said first temporal 
progression and said unknown spatial relationship between said at least first and second 
sequences with a sequential application of said unknown spatial relationship between 
said at least first and second sequences and at least one of said intra-sequence 
representation-to-representation transformations of said second temporal progression. 

1.40. A system according to any of claims 135 - 139 and "wherein equating 

includes equating up to a scale factor. 

141. A system according to any of claims 124 - 140 and wherein said 
intra-sequence representation-to-representation transformations include multiple simple 
motions taking place at least partially at different times. 

142. A system according to any of claims 124 - 141 and wherein said 
intra-sequence' representation-to-representation transformations include multiple 
combinations of multiple simple motions taking place at least partially at different 
times. 
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143. A system according to any of claims 124 - 142 and wherein said 
intra-sequence representation-to-representation transformations include multiple 
complex motions taking place at least partially at different times. 

144. A system according to any of claims 124 - 143 and wherein said 
intra-sequence representation-to-representation transformations include multiple 
combinations of multiple complex motions taking place at least partially at different 
times. . 

145. A system according to any of claims 89 - 144 and wherein said first and 
second temporal progressions comprise ordered intra-sequence 
representation-to-representation transformations at least some of which result from 
relative motion between sensors and a scene. 

146. A system according to any of claims 89- 145 and wherein said computation 
functionality uses multiple combinations of intra-sequence 
representation-to-representation transformations. 

147. A system according to any of claims 89 - 146 and wherein said spatial 
relationship between said at least first and second sequences of representations results 
from an acquisition relationship, between first and second sensors acquiring respective 
said at least first and second sequences, being fixed over time. 

148. A system according to any of claims 89 - 143 and wherein said spatial 
relationship between said at least first and second sequences of representations results 
from an acquisition relationship, between first and second sensors acquiring respective 
said at least first and second sequences, changes in a known way over time. 

149. A system according to claim 147 or claim 148 and wherein said acquisition 
relationship comprises: 

relative position; 
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relative orientation; and 

relative internal sensor parameters. 

150. A system according to claim 147 and wherein said acquisition relationship 
is known. 

151. A system according to claim 147 and wherein said acquisition relationship 
is not known. 

152. A system according to any of claims 89-151 and wherein said at least first 
and second sequences are acquired generally at the same time. 

1 53. A system according to any of claims 89 - 152 and wherein said at least first 
and second sequences are acquired generally at different times. 

154. A system according to any of claims 89 .- 153 and wherein said at least first 
and second sequences represent measurements from the same scene. 

155. A system according to any of claims 89- 154 and wherein said at least first 
and second sequences represent measurements from different portions of the same 
scene. 

1 56. A system according to any of claims 89 - 155 and wherein said at least first 
and second sequences represent measurements from different overlapping portions of 
the same scene. 

157. A system according to any of claims 89 - 156 and wherein said at least first 
and second sequences represent measurements from different non-overlapping portions 
of the same scene." 

1 58. A system according to any of claims 89 - 157 and wherein said at least first 
and second sequences represent measurements from different scenes. 
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159. A system according to any of claims 154 - 158 and wherein said scene is 
two-dimensional. 

160. A system according to any of claims 154 - 158 and wherein said scene is at 
least three-dimensional. 

161. A system according to any of claims 154 - 158 and wherein said scene is 
static. 

162. A system according to any of claims 154 - 158 and wherein said scene is 
dynamic. 

163. A system according to any of claims 154 - 162 and wherein said 
measurements are generally the same for each sensor. 

164. A system according to any of claims 154 - 162 and wherein said 
measurements are generally different for each sensor. 

165. A system according to any of claims 154 - 164 and wherein said 
measurements include at least one of : 

illumination, heat, radiance, electromagnetic radiation, color, distance, 
density, sound and speed. 

166. A system according to any of claims 89 - 165 and also comprising 
computational functionality includes functionality for employing at least one of said 
spatial and temporal relationships for sequence fusion. 

167. A system according to any of claims 89 - 166 and also comprising 
computational functionality includes functionality for employing at least one of said 
spatial and temporal relationships for alignment of sequences obtained at different 
zooms. 
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168. A system according to any of claims 89 - 167 and also comprising 
computational functionality includes functionality for employing at least one of said 
spatial and temporal relationships for surveillance. 

5 

169. A system according to any of claims 89 - 168 and also comprising 
computational functionality includes functionality for employing at least one of said 
spatial and temporal relationships for generation of wide-screen movies from multiple at 
least partially non-overlapping narrow field of view movies. 

10 

170. A system according to any of claims 89 - 169 and also comprising 
computational functionality includes functionality for employing at least one of said 
spatial and temporal relationships for image fusion. 

15 171. A system according to any of claims 89 - 170 and also comprising 

computational functionality includes functionality for employing at least one of said 
spatial and temporal relationships for integrating information contained in said at least 
first and second sequences. 

20 172. A system according to any- of claims 89 - 171 and also comprising 

computational functionality includes functionality for employing at least one of said 
spatial and temporal relationships for alignment of images obtained at different zooms. 

173. A system according to any of claims 89 - 172 and also comprising 
25 computational functionality includes functionality for employing at least one of said 

spatial and temporal relationships for comparing information contained in said at least 
first and second sequences. 

174. A system according to any of claims 89 - 173 and also comprising 
30 computational functionality includes functionality for employing at least one of said 

spatial and temporal relationships for finding differences between information contained 
in said at least first and second sequences. 

75 



WO 02/47031 



PCT/IL01/01120 



175. A system according to any of claims 89 - 174 and also comprising 
computational functionality includes functionality for employing at least one of said 
spatial and temporal relationships for finding differences between information contained 
in said at least first and second sequences relating to the same scene at different times. 

176. A system according to any of claims 89 - 175 and also comprising 
computational functionality includes functionality for employing at least one of said 
spatial and temporal relationships for integrating information contained in said at least 
first and second sequences and thereby providing an information output which exceeds 
limitations of individual sensors. 

177. A system according to any of claims 89 - 176 and wherein said 
computational functionality comprises: 

comparing properties of said first temporal progression and said second 
temporal progression. 
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